ClaudeBot: what it is and how to block or allow it
ClaudeBot is Anthropic's web crawler that collects content to enhance its generative AI models. Its robots.txt token is ClaudeBot. Anthropic says blocking it signals that your site's future content should be excluded from training datasets. It is separate from Claude-SearchBot and Claude-User, which have their own robots.txt tokens.
| Operator | Anthropic, for Claude |
|---|---|
| Purpose | AI training |
| robots.txt token | ClaudeBot |
| User agent | No full string documented. Match on the robots.txt token. |
| Follows robots.txt | Yes. Anthropic says its bots honor industry-standard robots.txt directives and supports the non-standard Crawl-delay extension. |
| Runs JavaScript | No |
| IP ranges | Published list |
| Documentation | Anthropic |
What ClaudeBot does
ClaudeBot collects web content that Anthropic uses to enhance the utility and safety of its generative AI models. It is the training crawler, separate from Claude-SearchBot and Claude-User.
Should you block ClaudeBot?
Treat ClaudeBot as a training choice, and block or allow it separately from Claude-SearchBot and Claude-User. Anthropic says that blocking ClaudeBot signals that your site's future content should be excluded from its training datasets.
User-agent: ClaudeBot
Disallow: /User-agent: ClaudeBot
Allow: /Rules that apply
- Rule 02: Treat training bots and search bots as separate robots.txt decisions
- Rule 03: Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows
- Rule 04: Ship primary content in the initial HTML; most AI crawlers don't execute JavaScript
Questions
Should I block ClaudeBot?
That is a policy choice about training use. Anthropic describes ClaudeBot as the crawler for model training, and says blocking it excludes your site's future content from training datasets.
How do I block ClaudeBot in robots.txt?
Add User-agent: ClaudeBot followed by Disallow: / to the top-level robots.txt of each subdomain you want to opt out. Anthropic also supports Crawl-delay if you only want to slow it down.
What is the ClaudeBot user agent string?
Anthropic's help article names the bot but does not publish a full user agent string. Match on the ClaudeBot token, and check source IPs against Anthropic's published list at claude.com/crawling/bots.json.
Does blocking ClaudeBot hide me from Claude search answers?
Anthropic documents search indexing under Claude-SearchBot and user-directed fetches under Claude-User. ClaudeBot is described for training only.
Sources
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic, Apr 7, 2026
- The rise of the AI crawler · Vercel, Dec 17, 2024
Checked against the operators' documentation every Monday. Last reviewed Nov 2, 2026.