Similarly, in the past CSS was rarely important for understanding the content on the page. But now, you can do some pretty amazing things with CSS. In some cases, if Google can't see the CSS on the page, it may mean that they are unable to see some important parts of the page's content.
As such, it is important to not block Google from seeing anything that is an important part of the content of your site. Doing so could negatively impact your rankings.
There are a couple of ways to tell whether you are blocking these files. First, to see if you have the message that Google sent out, go to Google Search Console (formerly Webmaster Tools) and click on Messages. Look for something that looks like this:
Here is the full text of the message:
Next, take a look at your robots.txt file. To do this, type the following into the url bar:
If you are running a site on WordPress, you may see something that looks like this:
So far, all of the messages that I have seen this week have been for WordPress sites. If your site is *not* WordPress, and you are not sure what files you are blocking improperly, it's probably best to speak to your developer or to an experienced SEO to determine what the best course of action is.
How do I fix the problem?
The solution is to remove the /wp-includes/ block from your robots.txt. If you are not comfortable with doing this, it is probably best to have someone with experience do this work for you as it is possible to do some serious damage to your search engine rankings if you do the wrong thing with your robots.txt file.
Still, for most sites on WordPress, the fix is pretty simple. If you have the Yoast SEO plugin installed, this will allow you to edit your robots.txt file. If you don't have it installed, you can safely install the plugin just for this purpose. Then, in your left sidebar, click on SEO and then Tools:
Now, select File Editor and robots.txt. Then you can remove the line that blocks wp-includes and save the file:
How to tell if you've fixed the problem
The next step is to go to Google Search Console and run a fetch and render as Googlebot. To do so, in the left column, select Crawl and then Fetch As Google. Leave the submit form blank to have Google check your homepage and click "Fetch and Render". It will take a minute or so to get results. You're probably going to see a green checkmark and "Partial":
Why partial? In most cases there are still going to be resources that Google is blocked from seeing. These will occur if you are using external resources. For example, you may see that your Adsense code is blocked. That's ok. I've seen a few people asking on Twitter if it's ok if Google fonts are blocked. This really shouldn't be an issue in my opinion. If, however, you use external resources that are vital to rendering content on your page, then these should not be blocked.
Ultimately, the best test is to click on the part where it says "Partial" and compare the two pictures you get. The one on the left is how Googlebot sees the page and the one on the right is how users see the page:
Scroll down to see your entire page. Are the two pictures the same? Are there parts of the page that Googlebot is not seeing? If it's just ads that are not being seen by Googlebot, that's ok. But, if parts of your page such as images, forms, etc. are not being seen then there still may be a problem.
What about security concerns?
— Gary Illyes (@methode) July 28, 2015
Also, it's still a good idea to keep the /wp-admin/ block in robots.txt. This will stop search engines from indexing your login page.
Any time Google sends out a crazy number of messages, there tends to be a lot of confusion. While an SEO professional likely knows what these messages mean, I am guessing that most small business owners who got this message were a little confused as to what to do. Hopefully this has helped, but if you have further questions, please ask them in the comments section and I'll see if I can help.
Google update newsletter
Want an update when Google makes a big algorithm change or other announcement? Sign up here!