What is crawl-delay? The robots.txt rule explained honestly

Crawl-delay is a rule in your robots.txt file that asks search engines to wait a number of seconds between two consecutive page requests. It is meant to prevent a crawler from overloading your server by requesting pages too quickly in a row. The honest nuance right up front: Google ignores crawl-delay completely, while Bing and Yandex do follow the rule. In this article you will read what crawl-delay actually does, which search engines act on it, and when it makes sense for your B2B site (spoiler: almost never).

What exactly is crawl-delay?

Crawl-delay is a directive you place in your robots.txt, the text file you use to give crawlers instructions. The syntax is simple:

User-agent: *
Crawl-delay: 10

This rule asks every crawler that respects it to wait at least ten seconds before requesting a next page. The thinking behind it: an aggressive bot pulling dozens of pages per second can temporarily take down a weak server. With crawl-delay you throttle that pace.

Important to know: crawl-delay never became an official web standard. It is a convention that some search engines follow and others do not. As a result, the same rule behaves very differently from one bot to the next.

Which search engines respect crawl-delay?

This is where the sting is, and exactly why crawl-delay so often ends up misconfigured.

• Google ignores crawl-delay. Google confirms in its robots.txt documentation that it does not support the rule. Googlebot determines its own crawl rate automatically, based on how quickly and cleanly your server responds. If your server responds slowly, Google dials back its own pace so it does not overload your site.
• Bing respects crawl-delay. Bingbot interprets the value as time windows: with Crawl-delay: 10, Bing fetches at most one page per ten-second window.
• Yandex respects crawl-delay. Yandex waits at least the specified number of seconds between two requests.

The practical conclusion: if you set a crawl-delay to slow Google down, nothing happens. Google simply does not read the rule. If you want to steer Googlebot, you do it through your server response and not through robots.txt.

Why Google picks its own pace

Google assumes it is smarter to adjust the crawl rate dynamically than to stick to a fixed number. The logic: if your server responds fast and healthily, Google can comfortably fetch a few more pages per visit. If Googlebot notices your server slowing down or returning error codes, it slows down on its own. That way Google protects your server without you having to set any number at all.

That fits with how crawl budget works: the faster your server responds, the more pages Google can fetch and index within the same time. A crawl-delay would work directly against that. You would force Google (if it read the rule at all) to crawl more slowly, which would make new or updated content land in the index later. For a site that runs on search traffic, that is the opposite of what you want.

When crawl-delay does (or does not) matter for B2B

We would rather say it honestly up front: for the vast majority of Belgian B2B sites, crawl-delay is unnecessary. You typically have a few dozen to a few hundred pages, not millions. Enough crawl traffic to make your server buckle simply does not happen with a normal brochure site or knowledge base.

Crawl-delay can be useful in a few edge cases:

• A very large site on weak, shared hosting, where less well-behaved bots (think of some SEO tools or scrapers) noticeably load your server.
• A spike in server load that demonstrably comes from crawl traffic from Bing or Yandex.

But even then, crawl-delay is usually a band-aid on a wooden leg. The real problem is your server capacity, not the crawlers. You solve that structurally much better with caching, a faster hosting environment and removing heavy queries. How that works, you can read in our explainer on improving server response time. That way you increase your findability instead of slowing it down.

If you really want to limit aggressive or unwanted bots, you work more precisely with blocks at server level or firewall rules than with a crawl-delay that only well-meaning crawlers read. Our SEO approach therefore never starts from a crawl-delay, but from the question of whether your server serves your most important pages fast enough to both visitors and crawlers.

How do you check your crawl behaviour?

Instead of blindly setting a crawl-delay, first look at what is actually happening. The crawl stats in Google Search Console show how many pages Google fetches per day and how your server responds to that. If the average response time rises, that is a signal to tackle your server, not to slow Google down. In your server logs you also see which bots make how many requests, so you know exactly whether a crawler is a problem or not.

Frequently asked questions about crawl-delay

Does crawl-delay work for Google? No. Google ignores the crawl-delay rule in robots.txt completely and determines its crawl rate automatically based on your server response. If you want to steer Googlebot, you do it through your server performance, not through crawl-delay.

Which search engines do respect crawl-delay? Mainly Bing and Yandex. Bing treats the value as time windows, Yandex as a minimum waiting time between requests. Other, smaller crawlers sometimes stick to it and sometimes do not.

Do I need crawl-delay for my B2B site? Almost never. Most B2B sites have too few pages to overload a server. You mostly risk important pages being picked up more slowly. Invest in a fast server instead.

How do I set crawl-delay if I still want it? In your robots.txt, under the right user-agent, you add a line such as Crawl-delay: 10. Then test in your server logs whether the desired effect occurs, and keep in mind that Google simply skips the rule.

Not sure whether your robots.txt is right?

Crawl-delay is a small detail that is often deployed to fix a problem that sits somewhere else. We look at the whole picture: does your server respond fast enough, are your commercial pages crawled smoothly, and does that deliver leads instead of noise? We are a small team that moves fast and tells you honestly when a setting adds nothing. Schedule your free intake