Robots.txt Template for WordPress
If you use WordPress, you should use a more extensive robots.txt file to optimize your website’s crawlability for SEO.
This boilerplate helps with crawl rate optimization by better utilizing the crawl budget that Google has assigned for your website, as it prevents Googlebot from attempting to access unimportant pages while ensuring they can still access critical files for rendering.
All you need to do is remove any lines that don’t apply, e.g., if you want Google to be able to access search pages, then remove the two lines beneath the search page’s comment. Finally, replace the Sitemap lines with your own sitemap file URLs.
User-agent: Googlebot # Allow files critical for rendering Allow: *.js Allow: *.css # Allow AJAX - Do Not Remove Allow: /wp-admin/admin-ajax.php # Prevent crawl-budget waste on search pages Disallow: /?s= Disallow: /search/ # Prevent private admin areas from being crawled Disallow: /wp-admin # Prevent duplicate /feed/ pages from being crawled Disallow: /*/feed/ # Prevent login page crawls etc Disallow: /wp-login.php # Prevent register page crawls etc Disallow: /wp-register.php # Prevent Trackback Neg SEO Disallow: /trackback/ User-agent: * # Allow AJAX - Do Not Remove Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin Disallow: /wp-login.php Disallow: /trackback/ Disallow: /wp-register.php # Add all sitemaps Sitemap: https://example.com/sitemap_index.xml Sitemap: https://example.com/post-sitemap.xml Sitemap: https://example.com/page-sitemap.xml Sitemap: https://example.com/category-sitemap.xml Sitemap: https://example.com/post_tag-sitemap.xml
Need help with a custom robots.txt for WordPress? Say hello to let me know what you need!