WordPress robots.txt File

Using robots.txt file on your WordPress built site? If not, you should.

How to create it? Easy, just open a Microsoft Notepad and create a text file and name it robots.txt

Then upload this to your WordPress installation.

Login to cPanel > File Manager > public_html (path may be different for your website setup) > upload robots.txt here.

When you need to provide directives so that you can better control (or even block) crawling of search engines like Google, here’s all you need to know about using robots.txt file for WordPress Content Management System.

It is absolutely important that you actually understand what these directive do, or else, you may see many Google Search Console Errors (such as URL’s blocked by robots.txt).

Google search engine before it indexes your website, it CRAWLS IT using a user-agent called Googlebot.

Other user-agents = search crawlers, web crawlers all mean the same thing. They basically request URL to see what’s on that URL

WordPress built site owners can use the robots.txt file to give further instructions about their website to Googlebot. This is called The Robots Exclusion Protocol. Below are 2 additional readings to expand your knowledge about Google and Website optimization.

• Introduction to robots.txt
• Create a robots.txt file

How to Create & Use robots.txt on WordPress (Video by RankYa)

WordPress robots.txt File Location

By default, when you install WordPress, there is NO robots.txt file on your installation. Although that is the case, WordPress will automatically create a virtual robots.txt file. Hence the confusion when you try to look for such file on your web server. Usually you would look here:

File Manager > public_html > robots.txt file

And then you don’t see anything because there is NO physical robots.txt file. That means, you need to create one like this:

WordPress robots.txt File Example 1

User-agent: * Disallow: Simply create a text file and call it robots.txt and then insert the above code and upload to your web server where your WordPress is installed. Above sample code says

Hey Googlebot you CAN access my entire website, I disallow you NOT

WordPress robots.txt File Example 2:

User-agent: Googlebot Disallow: /cgi-bin/ Disallow: /wp-admin/$ Disallow: */trackback/$ Disallow: /comments/feed* Disallow: /wp-login.php?* Allow: /*.js* Allow: /*.css* Allow: /wp-admin/admin-ajax.php Allow: /wp-admin/admin-ajax.php?action=* Allow: /wp-content/uploads/* User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/$ Disallow: */trackback/$ Disallow: /comments/feed* Disallow: /wp-login.php?* Allow: /*.js* Allow: /*.css* Allow: /wp-admin/admin-ajax.php Allow: /wp-admin/admin-ajax.php?action=* Allow: /wp-content/uploads/* #Sitemap: https://CHANGE/page-sitemap.xml #Sitemap: https://CHANGE/post-sitemap.xml #Sitemap: https://CHANGE/product-sitemap.xml

In the above robots.txt file example, all you need to change is the location of your sitemaps, and then remove the #tag as it is used as a comment. Just create a text file and call it robots.txt and then place the above code and upload to your web server where your WordPress is installed (once you change sitemap location). You can safely use the above code as it will help your WordPress site avoid many Google Search Console errors such as Server 5xx Errors.

Download robots.txt File Example

robots.txt (.zip format, simply unzip it)

WordPress robots.txt Not Updating?

Its usually because you are seeing the cached version of your website. You need to clear the cache of your internet browser.

WordPress robots.txt Yoast SEO Plugin?

Yoast SEO plugin for WordPress does create a robots.txt file for you and allows you to modify it from your WordPress dashboard. Simply use the sample codes above and Save changes to robots.txt file.

WordPress robots.txt Plugin?

Do NOT use plugins for using robots.txt file because creating one is simple as detailed above.

URL Blocked by robots.txt WordPress

What this means is that when Google tries to crawl your website / web pages, it sees something in the robots.txt file that says “Hey Google you are NOT allowed (hence DISallow) to crawl that URL” causing URL blocked by robots.txt file errors in Google search console.

How to Unblock URL blocked by robots.txt in WordPress

Simply use the instructions shown in WordPress robots.txt File Example 1 above as it will unblock any directives in your robots.txt file

How to Unblock URL blocked by robots.txt in WordPress

To avoid most issues with Google Search Console robots.txt blocking certain URL’s from your own WordPress built web site, understand the fact that Google’s CRAWLING and INDEXING Process is 2 completely different processes.

That means, just because you are using robots.txt file does NOT mean you are telling Google to NOT index certain parts of your website. To be able to control Google’s indexing certain parts, you need to use meta tags with noindex option. Example only for WordPress CMS <?php if (is_front_page()) : ?> <meta name="Googlebot" content="index"> <meta name="robots" content="index"> <!--this is front--> <?php elseif (is_search()) : ?> <meta name="Googlebot" content="noindex"> <meta name="robots" content="noindex"> <!--searchqueries now part of the main conditional logic--> <?php elseif (is_page('samplePageNameYouWantToNOINDEX')) : ?> <meta name="Googlebot" content="noindex"> <meta name="robots" content="noindex"> <?php elseif (is_singple('sampleBlogPostNameYouWantToNOINDEX')) : ?> <meta name="Googlebot" content="noindex"> <meta name="robots" content="noindex"> <?php endif; ?> Path to that file is Login to Web Hosting > File Manager > public_html > wp-content > themes > YourThemeName > header.php

Basically, locate header.php file and find < head > portion and modify above PHP code and then copy paste save, to control indexing of Google for certain parts of your WordPress site.

RankYa WordPress SEO Tip

• do NOT index tags so use is_tag()
• do NOT index attachment URL’s is_attachment (unless your website is about photography or photo sharing site)
• do NOT index paginated content is_single() && is_paged()