Crawl delay robots.txt rules: does Google respect them?
No. Google does not respect crawl-delay. Its robots.txt documentation lists only user-agent, allow, disallow and sitemap as supported fields and names crawl-delay as unsupported. Some other bots do honor it, including Anthropic's. To slow Googlebot under overload, return 503 or 429 temporarily, and never use 401 or 403 to limit crawl rate.
This no longer works. Google's robots.txt parser supports only user-agent, allow, disallow and sitemap, and explicitly says crawl-delay is not supported.
| Status | Evidence | Applies to | Retired |
|---|---|---|---|
| Retired | Official guidance | Not dated |
Why it stopped working
Google's robots.txt parser supports only user-agent, allow, disallow and sitemap, and explicitly says crawl-delay is not supported. Some other bots, including Anthropic's, do honor it.
What to do instead
- Let Google adjust crawl rate automatically, or return 503/429 temporarily under overload
- Keep crawl-delay only for bots that document support for it
Example
User-agent: ClaudeBot
Crawl-delay: 1Questions
Which bots support crawl-delay?
Anthropic documents support for the non-standard Crawl-delay extension and gives a one-second ClaudeBot example. Google does not support it. Keep crawl-delay only for bots that document support, and check each bot's documentation before relying on it.
How do I reduce Googlebot's crawl rate?
Let Google adjust its crawl rate automatically, and under real overload return 503 or 429 temporarily. Google says other 4xx codes have no effect on crawl rate, and warns against using 401 or 403 to limit it.
Does a crawl-delay line affect Googlebot at all?
No. Google treats crawl-delay as an unsupported field, so it has no effect on Googlebot. Googlebot follows the user-agent, allow, disallow and sitemap lines in the same file.
Sources
- Robots.txt introduction and guide · Google Search Central, Oct 23, 2026
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic, Apr 7, 2026
Change history
| Date | Change | What changed |
|---|---|---|
| Oct 30, 2026 | Added | Added to the rulebook. |
Checked against its sources every Monday. Only content changes are listed.