A noindex tag is a directive you add to a webpage that tells search engines to leave that page out of their index. It comes in two flavours: a meta robots tag in the page’s HTML head, or an X-Robots-Tag sent as an HTTP response header.
Both forms do the same job. They give you a clean way to say, “Crawl this if you want, but don’t show it in search results.” Used well, noindex keeps thin and useless pages out of Google while letting the pages you actually care about pull their weight.
How a Noindex Tag Actually Works
Googlebot requests the page, downloads the HTML or reads the response headers. If it sees a noindex directive, it processes the page but excludes it from the search index. If the page was previously indexed, Google drops it on the next crawl.
The meta tag version sits inside the <head>:
<meta name="robots" content="noindex">
The HTTP header version is useful for non-HTML files like PDFs:
X-Robots-Tag: noindex
One important nuance: the noindex directive only works if the crawler can actually reach the page and read it.
When to Use Noindex
- Thank-you and confirmation pages. Shown after a form fill or purchase. You don’t want them ranking.
- Internal search results pages. Google has been clear these tend to be poor experiences when surfaced.
- Paginated archives that add no value. Page 17 of your blog archive isn’t doing anyone favours.
- Thin tag and category pages. If a tag page has three posts and no original content, noindex it.
- Admin, login, and account pages. No reason for /wp-admin or /my-account to show up in search.
- Staging and development sites. If not behind HTTP authentication, noindex every page.
- Duplicate content variants. Filtered or sorted versions of product listings, print-friendly pages.
Noindex vs Robots.txt: They Are Not the Same
| Mechanism | What It Blocks | Effect on Indexing |
|---|---|---|
| robots.txt Disallow | Crawling (Googlebot won’t request the page) | Page can still appear in results if linked from elsewhere, just with no description |
| Noindex meta or header | Indexing (Googlebot crawls but excludes the page) | Page is removed from results entirely once Google sees the directive |
The key insight: robots.txt is a crawl instruction, not an index instruction. If you Disallow a URL in robots.txt, Google might still index it based on external links. Use noindex when you want a page crawlable but kept out of search results.
Common Mistakes That Will Bite You
1. Combining noindex with a robots.txt Disallow
This is the big one. If you Disallow a URL in robots.txt, Googlebot never fetches the page, which means it never sees your noindex tag. The page can sit in the index indefinitely. If you want a URL deindexed, you have to let Google crawl it long enough to read the noindex. The order matters.
2. Accidentally Noindexing Pages That Should Rank
The classic disaster: a developer launches a new site and forgets to uncheck the “Discourage search engines” box in WordPress, which slaps a noindex on every page. Or a theme update introduces a sitewide noindex by mistake. Always run a crawl after any significant change.
3. Forgetting to Remove Noindex When Launching
Staging sites get noindex tags. The problem is when they ship to production with those tags still in place. Build a launch checklist that includes “remove noindex” as a hard requirement.
How to Implement Noindex
Meta Tag
<meta name="robots" content="noindex, follow">
The “follow” part tells search engines they can still follow links on the page, which is usually what you want.
X-Robots-Tag HTTP Header
For PDFs, images, or bulk rules, set the header at the server level via .htaccess (Apache) or server config (Nginx):
X-Robots-Tag: noindex, follow
WordPress with Yoast or Rank Math
Both plugins give you a per-post toggle in the editor sidebar. Look for “Allow search engines to show this post in search results?” or “No Index.” They also let you noindex entire post types, taxonomies, or archive types from plugin settings.
How to Check That Noindex Is Working
- View source. Right-click, View Page Source, search the HTML for “noindex.”
- GSC URL Inspection. Paste the URL in Search Console. It’ll tell you whether Google detected a noindex on its last crawl. This is the source of truth.
If you’ve just added a noindex and the page hasn’t been recrawled, request indexing through URL Inspection to nudge Google. Deindexing usually happens within days.
The Bottom Line
Noindex is a precision tool. The most important rule: if you want a page deindexed, Google has to be able to crawl it. Don’t combine noindex with a robots.txt Disallow on the same URL, or you’ll end up with the exact opposite of what you wanted.
Audit your site once a quarter. Check that important pages are indexable and utility pages aren’t. A few minutes with a crawler and the URL Inspection tool will save you from indexing mistakes that quietly drain traffic for months.