Claude-SearchBot user agent: what it is and whether to allow it
Claude-SearchBot is Anthropic's crawler that navigates the web to improve the quality of Claude's search results. Its robots.txt token is Claude-SearchBot. Anthropic says blocking it stops indexing of your content for search and may reduce your visibility and accuracy in user search results. It is separate from ClaudeBot.
| Operator | Anthropic, for Claude search |
|---|---|
| Purpose | AI search |
| robots.txt token | Claude-SearchBot |
| User agent | No full string documented. Match on the robots.txt token. |
| Follows robots.txt | Yes. Anthropic says its bots honor industry-standard robots.txt directives and supports the non-standard Crawl-delay extension. |
| Runs JavaScript | Not documented |
| IP ranges | Published list |
| Documentation | Anthropic |
What Claude-SearchBot does
Claude-SearchBot navigates the web to improve the quality of search results for Claude users. It indexes content so it can be used in those results.
Should you block Claude-SearchBot?
Allow Claude-SearchBot if you want your pages to be found in Claude search results; it is separate from ClaudeBot. Anthropic says blocking Claude-SearchBot stops indexing of your content for search and may reduce your visibility and accuracy in user search results.
User-agent: Claude-SearchBot
Allow: /User-agent: Claude-SearchBot
Disallow: /Rules that apply
- Rule 01: Allow OAI-SearchBot, PerplexityBot and Claude-SearchBot in robots.txt if you want AI citations
- Rule 02: Treat training bots and search bots as separate robots.txt decisions
- Rule 03: Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows
Questions
What is the Claude-SearchBot user agent string?
Anthropic's help article names Claude-SearchBot but does not publish a full user agent string. Use the Claude-SearchBot token in robots.txt, and check IPs against claude.com/crawling/bots.json.
Should I allow Claude-SearchBot?
Anthropic says blocking it may reduce visibility in user search results, so allow it if you want Claude to be able to find and surface your pages.
Is Claude-SearchBot used for training?
Anthropic describes ClaudeBot as the training crawler and Claude-SearchBot as the search quality crawler. Block them separately according to your goals.
Does Claude-SearchBot respect robots.txt?
Yes. Anthropic says its bots honor industry-standard robots.txt directives, and it supports the non-standard Crawl-delay extension.
Sources
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic, Apr 7, 2026
Checked against the operators' documentation every Monday. Last reviewed Nov 2, 2026.