Google Search Console Index Coverage Report shows you the status of your WordPress site’s indexed pages. Or if Google search engine is having problems when it tries to index URLs but your web server where your WordPress is installed can’t handle Google’s request thus throws an exception 500 Internal Server Error.
In this blog post I’m creating for RankYa fans, I’ll show you how to fix common Server Errors 5xx WordPress Content Management System experiences due to the way WordPress is built and due to the way Google crawling and indexing works. Furthermore: I’m going to share tips for best practices so that your business website is optimized better.
What is Server Error 5XX
500 Internal Server Error is a common HTTP status code that tells you that there is something wrong INTERNALLY on the web server. For example: it could be 502 Bad Gateway or even 500 Internal Server Error. Meaning, since Web Server Error Responses can range between (500–599) depending on the problem. Search Console just adds couple of xx’s next to 5 without exactly telling website owners what the error actually is.
Video Lesson for Fixing Search Console Index Coverage WordPress 500 Internal Server Errors
Google Search Console Coverage Server Error (5xx)
Once again, although Google Search Console Coverage report does NOT tell us exactly what the error is, I can through confidence and experience tell you that your WordPress built site is responding to Google’s requests with 500 Internal Server Error.
Common WordPress URLs That Respond With 500 Server Response Code
- wp-includes
- wp-content/plugins
- wp-content/theme/WordPressTheme/inc/sample.php
- wp-admin
Above locations are folder structures of WordPress CMS, and the way WordPress works conflicts with Googlebot. Meaning, wp-includes folder is needed for WordPress built site to function. And yet, there is nothing in that folder that Google needs to index. Same is true with wp-admin folder which is only needed for administrator of WordPress (your WordPress Dashboard).
Since google being an efficient search engine following links it finds, it usually finds URLs pointing to parts of WordPress internal folder or files URLs such as:
example.co.uk/wp-content/plugins/elementor/widgets/main.php
example.co.uk/wp-content/theme/avada/inc/index.php
example.co.uk/wp-content/plugins/gallery/configuration/config-main.php
That is why Google Search Console Index Coverage report will show Server error (5xx) for WordPress built websites.
How to Fix Server Error 5xx for WordPress?
Think of troubleshooting this problems in two different ways. Part 1 is you simply ignore the error because it could actually be a temporary problem (e.g. Google tried to crawl the URL while server was in reboot process). Simply keep an eye on it till next time Googlebot requests the URLs it found to respond with 5xx Server Error.
Fixing 5xx Internal Server Errors in WordPress
There are many different types of WordPress setups, therefore, I’m going to show you all the ways to ensure your WordPress website does not deal with 5xx Server Errors.
Fixing Server 5xx Errors Using .htaccess File
Simply login to your web server cPanel and locate your WordPress installation folder. Then, browse in to wp-includes, press create +File and name that file .htaccess
Copy/Paste below code in to .htaccess file and save
<FilesMatch "\.(php|css|js)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
Header set X-Robots-Tag "noindex"
Follow the same process but this time create another .htaccess and place it in wp-admin folder.
Fixing Server 5xx Errors Using robots.txt File
Simply create a new file can name it as robots.txt and place these lines of directives to ensure Google does NOT request the URLs with errors
Ensure WordPress XML Sitemaps are Setup Correctly to Avoid Server Errors
Most WordPress built websites aren’t optimized correctly, nor the correct XML sitemaps are submitted to Google Search Console. For example: WordPress Date or Tag based archive URL’s should not be indexed by Google nor included in XML sitemaps.
Best XML Sitemaps Settings for Avoiding 500 Internal Server Error for WordPress
Since most WordPress built websites use Yoast SEO plugin XML functionality, keep in mind to ONLY submit the XML sitemap you intend to rank in Google. For example: submit
https://www.example.com/post-sitemap.xml
https://www.example.com/page-sitemap.xml
Operating WooCommerce? Then, submit XML sitemaps through Google Search Console like so:
https://wwwwww.example.com/post-sitemap.xml
https://www.example.com/page-sitemap.xml
https://www.example.com/product-sitemap.xml
Best Practices for WordPress NO INDEX Meta Tag Settings
The best way to ensure Search Console doesn’t bother you with Index Coverage errors is to optimize the header.php file specifically setting noindex directives to certain parts of your WordPress site. For example: when a valued client hires RankYa, then, I analyze the entire website and work out what URLs should be indexed by Google. Upon which, I simply code header.php file using Conditional Tags using PHP
RankYa SEO Tip for WordPress Content Management System Experiencing Search Console Issues
Do not worry too much about URLs which your website isn’t interested in ranking in Google. Always ask “which parts of my WordPress website has unique original content. Then, ensure Google is aware only of those URLs and forget about the rest”.
As you can see, its easy to remedy Server error (5xx) for Index Coverage report for typical WordPress setup. Having said all this, keep in mind that if the problem persists, your server may be setup incorrectly or it could be just bad web hosting on shared server.
The in-depth guide to the 5XX server error I’d love to read again & again to absorb the knowledge.
Really love this code
Header set X-Robots-Tag “noindex”
Header set X-Robots-Tag “noindex”
Thanks for guiding us in the SE world.