Crawl Stats in Search Console: Reading and Using Your Crawl Budget

The Crawl Stats report in Google Search Console shows how often and how smoothly Google has crawled your website over the past 90 days. You will find it tucked away under Settings, and it reveals whether there are server errors, how fast your pages load and whether Google actually reaches your important pages. In this article you will read where the report sits, how to read it and how to use it, along with your crawl budget, to diagnose crawl issues on a B2B site.

What are crawl stats in Search Console?

Crawling is the process where Googlebot fetches your pages so they can be indexed and shown in the search results, as described in Google’s documentation on crawling and indexing. The Crawl Stats report shows how that went over the last 90 days: how many requests Google made, which responses your server gave and which problems the bot ran into.

The report answers questions like: when was my site last crawled, how many requests did Google make, and how many errors did the bot encounter. It is part of Google Search Console, the free tool that connects you to the SEO work we do for you.

Honest advice: not everyone needs this report. According to Google, websites with more than 1,000 pages find the data most valuable. If you have a small brochure site of twenty pages, you will solve crawl issues faster with a technical SEO lens than with this dashboard, something we include as standard in our SEO. We are a B2B agency, not a webshop factory, and we say it honestly when a tool is overkill for your situation.

Where do you find the Crawl Stats report?

The report is easy to miss, because it does not sit with the regular reports in the sidebar. Here is how to find it:

1. Click Settings in the bottom left of Search Console.
2. Scroll to the Crawling section.
3. Click Open report next to Crawl stats.

That is all. On the next page you see the full report with all the charts.

Note one nuance that Google mentions itself: the report contains most crawl requests, but not all of them. If you compare it with your server logs, small differences can show up. So use it as a directional picture, not as accounting down to the unit.

How do you read the charts?

The report looks messy at first glance. These are the parts that matter:

• Total crawl requests: how often Google visited your site, successful and failed combined.
• Total download size: how much data Google fetched. A high average can point to heavy files or a lot of fetched content.
• Average response time: how fast your server responds. Lower is better and points to a faster server, which enables faster crawling and indexing.
• Host status: whether Google had problems with your robots.txt, DNS or server connectivity. A green checkmark means no problem, an exclamation mark asks for attention.
• Crawl responses: all server codes (200, 301, 404 and so on) with the percentage per type.
• File type: the split between HTML, images, JavaScript, video and the rest.
• Crawl purpose: whether Google fetches new pages (discovery) or revisits known pages (refresh).
• Googlebot type: which of Google’s crawlers visit your site and how often.

You do not have to check every chart daily. The value sits in the deviations: a sudden spike in errors, a slow server or a crawl purpose that leads nowhere.

Which crawl problems does it spot?

The report is strong at detecting three kinds of problems.

Server errors. If you see a lot of 5XX responses, Google cannot reach your site properly. This is often the most serious and the hardest to solve. Contact your hosting provider to check whether there is an outage or a faulty configuration. For a B2B site that runs on leads, a server that pushes Googlebot away is direct revenue loss.

Page errors. Many 404s harm the user experience and nibble away at your presence in the search results. Even well maintained sites have a few, but at large numbers you have to step in. Download the list of 404 pages that Google offers and set up 301 redirects to a fitting alternative. If a page used to rank on a valuable keyword, consider rebuilding the content.

Crawl budget problems. Your crawl budget is the time and resources Google spends on your site. If you have big swings in your crawl frequency, hundreds of pages that get crawled but not indexed, or a low crawl rate for the size of your site, there is a problem here. You solve it by making crawling easier: a flatter site structure, more internal links so no page sits more than three clicks from your homepage, fixing broken links and improving your site speed. A structured SEO audit brings these loose threads together.

How do you use crawl data to rank better?

The report does not only help you find problems, it also steers your optimisation.

With the crawl purpose report you see whether Google picks up your new content. Most crawls are refreshes, even if you add pages daily. But if the share of discovery crawls drops very low, check whether your sitemap is still active and up to date. New pages that never get discovered simply do not deliver leads.

The average response time reveals speed problems. Slow response times mean Google struggles to load your content, something you investigate further with a technical lens and with Core Web Vitals. And if you improve your site (smaller images, less CSS), you see it back in the report: the total download size drops and the response time improves. That way you measure the effect of your technical work instead of guessing.

Here is the Customer Impact way of thinking: a high number of crawls is a vanity metric. What counts is whether Google reaches and indexes your commercial pages, your service pages and your strong content. That is where your customers and your revenue sit, not in a pretty chart.

Frequently asked questions about crawl stats

Where is the Crawl Stats report in Search Console? Under Settings in the left sidebar. Scroll to the Crawling section and click Open report next to Crawl stats.

What exactly does the Crawl Stats report show? How Googlebot approached your site: the total number of crawl requests, the total download size, the average response time, the host status and the crawl responses per server code, over the past 90 days.

How do I check my crawl budget in Google Search Console? You estimate it by looking at the average number of daily crawl requests in the report. A higher average indicates that Google wants to crawl your site more often. Remember that many factors influence your crawl budget, not just the number of requests.

Does my small B2B site need this report? Often not intensively. According to Google, the report is most useful for sites with more than 1,000 pages. On a small site you track down crawl issues faster with a technical SEO checklist.

Need help with your crawl diagnosis?

Crawl stats are a fine early warning, but the real gain sits in what you do with them: solving server errors, cleaning up your structure and steering on the pages that bring in customers. We are a small team that moves fast and gives honest advice, without jargon and without unnecessary tools. Want to know whether your site is getting in Google’s way?

Book your free intake