Claude-User user agent: what it is and whether to allow it
Claude-User is the Anthropic user agent that accesses a website when a Claude user asks a question requiring web content. Its robots.txt token is Claude-User. Anthropic says blocking it stops retrieval for user queries and may reduce your visibility in user-directed web search. Unlike some competitors, Anthropic says its bots honor robots.txt.
| Operator | Anthropic, for Claude |
|---|---|
| Purpose | User-triggered |
| robots.txt token | Claude-User |
| User agent | No full string documented. Match on the robots.txt token. |
| Follows robots.txt | Yes. Anthropic says its bots honor industry-standard robots.txt directives, including Claude-User, which differs from OpenAI and Perplexity's user-triggered fetchers. |
| Runs JavaScript | Not documented |
| IP ranges | Published list |
| Documentation | Anthropic |
What Claude-User does
Claude-User accesses a site when a Claude user asks a question that requires web content. It fetches pages on demand to support that user rather than crawling broadly.
Should you block Claude-User?
Allow Claude-User if you want Claude to be able to read your pages when a user asks about them. Anthropic says blocking Claude-User stops retrieval of your pages for user queries and may reduce your visibility in user-directed web search.
User-agent: Claude-User
Allow: /User-agent: Claude-User
Disallow: /Rules that apply
- Rule 01: Allow OAI-SearchBot, PerplexityBot and Claude-SearchBot in robots.txt if you want AI citations
- Rule 11: Don't rely on robots.txt to stop ChatGPT-User or Perplexity-User; they fetch on a user's behalf
- Rule 03: Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows
Questions
What is the Claude-User user agent string?
Anthropic's help article names Claude-User but does not publish a full user agent string. Use the Claude-User token in robots.txt and check IPs against claude.com/crawling/bots.json.
Does Claude-User respect robots.txt?
Anthropic says its bots honor industry-standard robots.txt directives. Its article does not carve out an exception for Claude-User.
What happens if I block Claude-User?
Anthropic says retrieval for user queries stops, which may reduce your visibility in user-directed web search.
Is Claude-User the same as ClaudeBot?
No. ClaudeBot collects content for model training. Claude-User fetches pages when a Claude user asks a question that needs web content.
Sources
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic, Apr 7, 2026
Checked against the operators' documentation every Monday. Last reviewed Nov 2, 2026.