Robots.txt Validator

Robots.txt Best Practices

The robots.txt file tells search engine crawlers which pages or sections of your site they can or cannot request. It lives at the root of your domain (e.g. https://example.com/robots.txt) and follows a simple text-based protocol.

Valid Directives

• User-agent — Specifies which crawler the following rules apply to. Use * for all bots.
• Disallow — Tells the bot not to crawl the specified path. An empty value means "allow all."
• Allow — Overrides a Disallow for a more specific path (supported by Google and Bing).
• Sitemap — Points to your XML sitemap. Must be an absolute URL. Can appear anywhere in the file.
• Crawl-delay — Requests a delay (in seconds) between crawler requests. Not supported by Google, but used by Bing and others.
• Host — Specifies the preferred domain version. Used mainly by Yandex.

Common Mistakes to Avoid

• Using Disallow: / blocks all crawling — your pages will not be indexed.
• Forgetting the User-agent directive before Disallow or Allow rules.
• Using relative URLs in Sitemap directives — they must be absolute (start with http).
• Blocking CSS/JS files that search engines need to render your pages.
• Relying on robots.txt for security — it is publicly readable and not an access control mechanism.

Wildcard Patterns

Google and Bing support * (match any sequence) and $ (end of URL) in path patterns. For example, Disallow: /*.pdf$ blocks all PDF files.