WordPress Robots.txt: SEO Best Practices and Setup Guide

The robots.txt file controls which parts of your WordPress site search engine crawlers can access. Configured correctly, it improves crawl efficiency, prioritizes important pages, and reduces unnecessary server load. Configured incorrectly, a single misstep can hide your entire site from search engines.

Best Robots.txt for WordPress

Here is a solid, minimal WordPress robots.txt configuration:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml

Key notes:

• Keep CSS and JavaScript files crawlable — do not block /wp-content/ or /wp-includes/
• Robots.txt controls crawling, not indexing. Use noindex meta tags or password protection for truly private pages
• The file must live at https://yourdomain.com/robots.txt
• The Sitemap directive is supported by all major search engines and recommended

What Robots.txt Does

Robots.txt serves several important functions:

• Manages crawl traffic: Prevents crawlers from hitting pages that waste crawl budget
• Saves server resources: Reduces load from bot crawling on resource-intensive pages
• Prioritizes crawl budget: Focuses crawler attention on your most important pages
• Sitemap integration: Points crawlers to your XML sitemap for better page discovery

Setting Up Robots.txt in WordPress

Default WordPress Behavior

WordPress creates a virtual robots.txt file automatically. By default, it blocks /wp-admin/ and allows everything else. This default is fine for most sites, but you can customize it for better control.

Method 1: SEO Plugin (Recommended)

Both Yoast SEO and Rank Math include a robots.txt editor in their settings. This is the easiest approach:

1. Navigate to your SEO plugin’s settings (e.g., Yoast → Tools → File Editor)
2. Edit the robots.txt content directly in the browser
3. Save changes — the plugin creates or updates the physical file

Method 2: Manual File Upload

Create a text file named robots.txt and upload it to your WordPress root directory via FTP or your hosting file manager.

What to Block in WordPress Robots.txt

• /wp-admin/ — WordPress admin area (always block, except admin-ajax.php)
• Internal search results — Block /search/ or /?s= to prevent thin content crawling
• Login pages — /wp-login.php does not need crawling
• Tag archives (optional) — If your tag pages are thin, consider blocking or noindexing them
• Author archives (optional) — Single-author sites often do not need author archives crawled

What NOT to Block

• /wp-content/ — Contains your CSS, JavaScript, and images. Blocking this prevents Google from rendering your pages correctly.
• /wp-includes/ — Contains core WordPress scripts. Must be crawlable.
• Your sitemap — Make sure your sitemap URL is accessible, not blocked.
• Important content pages — Never accidentally block pages you want indexed.

Common Robots.txt Mistakes

• Blocking CSS/JS: Google cannot render your pages properly, harming mobile-first indexing
• Using Disallow: / — Blocks your entire site from all crawlers. Only use for staging sites.
• Confusing crawling with indexing: Robots.txt blocks crawling, not indexing. Pages can still appear in search results without being crawled if other sites link to them.
• Forgetting the sitemap: Always include your sitemap URL in robots.txt
• Not testing changes: Always validate your robots.txt in Google Search Console’s robots.txt tester after making changes

Testing Your Robots.txt

Use Google Search Console’s URL Inspection tool or the robots.txt tester to verify your file is working correctly. Check that important pages are not accidentally blocked and that blocked pages are intentionally restricted.