Robot Selector
Indexing
Following
Snippets
Images
Other
Meta Robots Tag Output
How to use the Meta Robots Tag Generator
The meta robots tag is how you tell Google whether to index a page and follow its links. Unlike robots.txt (which controls crawling), meta robots controls what happens after crawl — index/noindex, follow/nofollow, archive/noarchive.
Pick the indexing rule
index (default) or noindex. Use noindex for thin content, login pages, internal search results, faceted-nav URLs you don't want competing with main pages.
Pick the link-following rule
follow (default) or nofollow. nofollow tells Google not to pass PageRank through links on the page. Rare on full pages — usually only on user-generated content like forums.
Add advanced directives if needed
noarchive (no cached version), nosnippet (no description in SERPs), noimageindex (don't index images), max-snippet:140, max-image-preview:large.
Place in or as HTTP header
Standard placement is <meta name="robots"> in the page head. For non-HTML files (PDFs), use the X-Robots-Tag HTTP header instead.
Why the meta robots tag is your indexing-control toolkit
Every URL on your site falls into one of three buckets: must rank, can rank, must not rank. The meta robots tag is how you tell Google which bucket each URL is in.
When to use noindex
- Internal search results — they create infinite duplicate content.
- Faceted nav / filtered category pages — same products in different orders.
- Thank-you / confirmation pages — post-purchase or post-form pages with no value to organic search.
- Login, cart, account pages — user-state-dependent, not relevant to search users.
- Low-quality archive pages — date archives, tag pages with 1-2 posts.
- Staging or test pages accidentally exposed — emergency control until you can take them down.
noindex vs robots.txt vs canonical
Three ways to keep a URL out of search, each different:
- robots.txt — blocks crawl. Page can still index if linked externally. Saves crawl budget.
- noindex — blocks indexing. Page is crawled but won't appear in results.
- canonical — consolidates ranking signals to a different URL. Both URLs remain crawlable; only the canonical indexes.
The robots.txt + noindex trap
If you block a page in robots.txt, Google can't fetch it to see the noindex tag. Pages blocked from crawl can still index if linked externally. The fix: allow crawl in robots.txt while applying noindex in the meta tag. Once Google sees the noindex, it removes the page from the index — then you can safely add the robots.txt block to save crawl budget.
Frequently asked questions
What's the difference between robots.txt and meta robots?
robots.txt controls crawling — whether Googlebot is allowed to fetch the URL. Meta robots controls indexing — whether Google should include a fetched URL in search results. They're separate layers; you often use both.
What's the default if I don't set a meta robots tag?
index, follow. Pages without an explicit meta robots tag are indexed and have their links followed. You only need to add the tag when you want something other than the default.
Can I use meta robots on PDFs or images?
Not directly — PDFs and images don't have HTML <head> sections. Use the X-Robots-Tag HTTP response header instead. Set it to X-Robots-Tag: noindex on the file's response.
Will noindex remove a page from Google immediately?
No — Google has to crawl the page again to see the new tag. Recrawl typically takes a few days for popular pages, weeks for low-traffic ones. To accelerate, request indexing in Search Console URL Inspection — it forces a recrawl within hours.
What's the difference between nofollow and noindex?
noindex says "don't include this page in search results." nofollow says "don't pass PageRank through links on this page." You can combine: noindex, nofollow means the page won't index AND its links won't transfer authority.