Treat training bots and search bots as separate robots.txt decisions
| Impact | Evidence | Applies to | Last checked |
|---|---|---|---|
| High | Official guidance | Google, ChatGPT, Gemini, Claude | Oct 30, 2026 |
Why it matters
GPTBot, ClaudeBot and the Google-Extended token control model training, while OAI-SearchBot, Claude-SearchBot and Googlebot control search visibility. The vendors document each setting as independent, so you can opt out of training and still be cited. A blanket block of every AI user agent usually costs citations for no gain.
How to do it
- List every AI user agent in your robots.txt and label it training, search or user-triggered
- Decide on training (GPTBot, ClaudeBot, Google-Extended) separately from search (OAI-SearchBot, Claude-SearchBot, PerplexityBot)
- Never block Googlebot to avoid AI features; that removes you from Google Search entirely
Sources
- Overview of OpenAI crawlers · OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic, Apr 7, 2026
- Google's common crawlers (Google-Extended) · Google Search Central
Change history
| Date | Change | What changed |
|---|---|---|
| Oct 30, 2026 | Added | Added to the rulebook. |
Checked against its sources every Monday. Only content changes are listed.