Robot Selector
Indexing
Following
Snippets
Images
Other
Meta Robots Tag Output
How to use the Meta Robots Tag Generator
The meta robots tag is how you tell Google whether to index a page and follow its links. Unlike robots.txt (which controls crawling), meta robots controls what happens after crawl — index/noindex, follow/nofollow, archive/noarchive.
Pick the indexing rule
index (default) or noindex. Use noindex for thin content, login pages, internal search results, faceted-nav URLs you don't want competing with main pages.
Pick the link-following rule
follow (default) or nofollow. nofollow tells Google not to pass PageRank through links on the page. Rare on full pages — usually only on user-generated content like forums.
Add advanced directives if needed
noarchive (no cached version), nosnippet (no description in SERPs), noimageindex (don't index images), max-snippet:140, max-image-preview:large.
Place in or as HTTP header
Standard placement is <meta name="robots"> in the page head. For non-HTML files (PDFs), use the X-Robots-Tag HTTP header instead.
Why the meta robots tag is your indexing-control toolkit
Every URL on your site falls into one of three buckets: must rank, can rank, must not rank. The meta robots tag is how you tell Google which bucket each URL is in.
When to use noindex
- Internal search results — they create infinite duplicate content.
- Faceted nav / filtered category pages — same products in different orders.
- Thank-you / confirmation pages — post-purchase or post-form pages with no value to organic search.
- Login, cart, account pages — user-state-dependent, not relevant to search users.
- Low-quality archive pages — date archives, tag pages with 1-2 posts.
- Staging or test pages accidentally exposed — emergency control until you can take them down.
noindex vs robots.txt vs canonical
Three ways to keep a URL out of search, each different:
- robots.txt — blocks crawl. Page can still index if linked externally. Saves crawl budget.
- noindex — blocks indexing. Page is crawled but won't appear in results.
- canonical — consolidates ranking signals to a different URL. Both URLs remain crawlable; only the canonical indexes.
The robots.txt + noindex trap
If you block a page in robots.txt, Google can't fetch it to see the noindex tag. Pages blocked from crawl can still index if linked externally. The fix: allow crawl in robots.txt while applying noindex in the meta tag. Once Google sees the noindex, it removes the page from the index — then you can safely add the robots.txt block to save crawl budget.
Every meta robots directive, and what it actually does
The generator above outputs all of these. This is what each one means, which engines honour it, and when it's the right call.
| Directive | What it does | Support |
|---|---|---|
index |
Allows the page into the index. This is the default — you never need to write it, and writing it does not override anything. | All engines |
noindex |
Keeps the page out of search results. The page is still crawled and its links still followed unless you also say nofollow. |
All engines |
follow |
Crawls and passes signals through the page's links. Also the default. | All engines |
nofollow |
Stops signals flowing through every link on the page. Blunt — for single links, use rel="nofollow" on the anchor instead. |
All engines |
noarchive |
Suppresses the cached copy in results. No effect on ranking. | Google, Bing |
nosnippet |
Removes the description text and the video preview. Usually costs you clicks — reach for max-snippet before this. |
Google, Bing |
max-snippet:[n] |
Caps the snippet at n characters. 0 means no snippet, -1 means no limit. |
|
max-image-preview:[setting] |
none, standard, or large. large is what you want for anything visual — it's required for image thumbnails in Discover. |
|
max-video-preview:[n] |
Caps video preview length in seconds. 0 yields a static image, -1 is unlimited. |
|
noimageindex |
Keeps images on the page out of Google Images. Does not apply to images that other pages also host. | |
notranslate |
Suppresses the translation prompt in results. Useful for brand or legal copy that must not be machine-translated. | |
unavailable_after:[date] |
Drops the page from results after a date (RFC 822 / ISO 8601). Built for event pages and time-boxed offers. | |
none |
Shorthand for noindex, nofollow. Rarely worth the ambiguity — write both out. |
Targeting one crawler at a time
Swap name="robots" for a specific user agent and the rule applies only to that bot. name="googlebot" targets Google, name="bingbot" targets Bing. A bot-specific tag overrides the generic one for that bot, so this pair leaves the page indexable everywhere except Google:
<meta name="robots" content="index, follow"><meta name="googlebot" content="noindex">
The same applies to the AI crawlers — GPTBot, ClaudeBot, PerplexityBot — though those are more reliably handled in robots.txt, since a meta tag only takes effect once the page has already been fetched.
When to use X-Robots-Tag instead
A meta tag only works in an HTML <head>. For anything that isn't HTML — PDFs, images, CSVs, generated downloads — the same directives travel as an HTTP response header, and they accept identical values.
X-Robots-Tag: noindex, nofollow
Apache
Drop this in .htaccess to keep every PDF on the site out of the index:
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>
Nginx
location ~* \.pdf$ { add_header X-Robots-Tag "noindex, nofollow"; }
The header is also the cleaner option when the rule is dynamic — expiring job listings, out-of-stock products, gated content — because your application can set it per response without touching the template.
Five mistakes that cost real traffic
1. Shipping a staging noindex to production
The single most expensive SEO accident there is. A site-wide noindex set on staging gets promoted with the release, and rankings collapse over the following days as Google recrawls. If traffic falls off a cliff with no algorithm update to blame, check this first — view source on the live site before you check anything else.
2. Blocking in robots.txt and expecting deindexation
Covered above, and worth repeating because it's so common: crawl-blocked pages can still appear in results, just without a description. Crawl must be allowed for the noindex to be read.
3. Using nofollow to sculpt PageRank
Google stopped honouring this as a redistribution mechanism years ago. Signals that would have flowed through a nofollowed link are simply lost, not redirected elsewhere on the page.
4. noindex on a page that other pages canonicalise to
Conflicting instructions: the canonical says "consolidate here", the noindex says "drop this". Google resolves the conflict unpredictably and may drop the whole cluster. Pick one — canonical for duplicates, noindex for pages that should not rank at all.
5. Leaving max-image-preview at the default
Not an error so much as a missed opportunity. Without max-image-preview:large, publishers lose the large thumbnail in Discover and image-rich results. It costs one directive.
Frequently asked questions
What's the difference between robots.txt and meta robots?
robots.txt controls crawling — whether Googlebot is allowed to fetch the URL. Meta robots controls indexing — whether Google should include a fetched URL in search results. They're separate layers; you often use both.
What's the default if I don't set a meta robots tag?
index, follow. Pages without an explicit meta robots tag are indexed and have their links followed. You only need to add the tag when you want something other than the default.
Can I use meta robots on PDFs or images?
Not directly — PDFs and images don't have HTML <head> sections. Use the X-Robots-Tag HTTP response header instead. Set it to X-Robots-Tag: noindex on the file's response.
Will noindex remove a page from Google immediately?
No — Google has to crawl the page again to see the new tag. Recrawl typically takes a few days for popular pages, weeks for low-traffic ones. To accelerate, request indexing in Search Console URL Inspection — it forces a recrawl within hours.
What's the difference between nofollow and noindex?
noindex says "don't include this page in search results." nofollow says "don't pass PageRank through links on this page." You can combine: noindex, nofollow means the page won't index AND its links won't transfer authority.