New Set of Crawl Errors Data in GWT

When doing onsite audits and figuring out technical issues that need to be fixed we first take a look at what Google Webmaster Tools has to say, and then do the manual work. Well, the first part got a lot easier as Google announced a new set of crawl errors data in GWT[1].

What is more the section with crawl errors is now better structured, namely you can see errors divided into two types, site wide errors and URL errors. Like the names imply, site errors are errors that affect the entire site starting from robots.txt, web server issues, DNS issues etc… While URL errors pint point a specific URL on your website.

crawl errors

Once you access you GWT, under Diagnostics you will find a tab called Crawl errors, once you click on it you will see crawl errors divided into site errors at the top, separated into DNS, Server Connectivity and Robots.txt Fetch. Underneath will be URL errors with specifics about server errors, access denied, not found, not followed, soft 404s and other.

Creating separate data was a smart and long awaited move from Google, like they said:

“We used to report these errors by URL, but that didn’t make a lot of sense because they aren’t specific to individual URLs—in fact, they prevent Googlebot from even requesting a URL!”

And in case these errors persist Google will start sending you notifications about them.

Another great addition is that you can now sort the errors by categories and priority. No more need to manually scroll through the errors or search your exports to find the most important ones. You switch between categories like soft 404s, not found and others and also arrange them by priority. Even better, now you can check them as fixed and let Google know about it instead of waiting for Google to pick it up by itself.

crawl errors filter

Next step is even better; you can click on each of the URLs in your errors section and see more details about the errors and the URL in question like last crawled, first detected, explanation of error details, errors in sitemaps and the pages that link to the errors URL.

error details

When I saw the priority tab the first question that popped up was how do they determine the priority, and they answered it quite well:

“We’ve ranked the errors so that those at the top of the priority list will be ones where there’s something you can do, whether that’s fixing broken links on your own site, fixing bugs in your server software, updating your Sitemaps to prune dead URLs, or adding a 301 redirect to get users to the “real” page. We determine this based on a multitude of factors, including whether or not you included the URL in a Sitemap, how many places it’s linked from (and if any of those are also on your site), and whether the URL has gotten any traffic recently from search.”

The errors marked as fixed are removed from the priority list, unless Google finds the same errors during the next crawl. Overall these are amazing improvements to GWT that will help any webmaster keep track of site errors and keep the onsite technical issues to a minimum. I saw bravo Google, you deserve credit for this one.



Zac is a link building specialist and a seasoned SEO professional. He manages a team of link builders and actively promotes some of Australia's biggest brands. Zac is an active blogger and also maintains Dejan SEO's news and updates section.

More Posts - Website