WordPress Robots.txt Boilerplate

Last modified: Friday, Mar 20, 2020

This is v1.1 of my robots.txt template for wordpress!


# WordPress Robots.txt Boilerplate v1.1 by Daniel Cuttridge
# https://danielcuttridge.com/wordpress-robots-boilerplate/
# UTF-8 BOM Tested & Approved

User-agent: Googlebot

# Allow files critical for rendering
Allow: *.js
Allow: *.css

# Allow AJAX - Do Not Remove
Allow: /wp-admin/admin-ajax.php


# Prevent crawl-budget waste on search pages
Disallow: /?s=
Disallow: /search/
# Prevent private admin areas from being crawled
Disallow: /wp-admin
# Prevent duplicate /feed/ pages from being crawled
Disallow: /*/feed/
# Prevent login page crawls etc
Disallow: /wp-login.php
# Prevent register page crawls etc
Disallow: /wp-register.php
# Prevent Trackback Neg SEO
Disallow: /trackback/


User-agent: *

# Allow AJAX - Do Not Remove
Allow: /wp-admin/admin-ajax.php

Disallow: /wp-admin
Disallow: /wp-login.php
Disallow: /trackback/
Disallow: /wp-register.php

# Add all sitemaps
Sitemap: https://candykeyboards.com/sitemap_index.xml
Sitemap: https://candykeyboards.com/post-sitemap.xml
Sitemap: https://candykeyboards.com/page-sitemap.xml

Update note: 3rd June 2020 - Changed JS and CSS rules to add * prefix in line with new recommendations from Google's Ligthouse Audit.

Update note: 3rd June 2020 - Made /feed/ implicit to only after an existing directory or URL.



Share this post 👍