Don't rely on robots.txt to stop ChatGPT-User or Perplexity-User; they fetch on a user's behalf
| Impact | Evidence | Applies to | Last checked |
|---|---|---|---|
| Low | Official guidance | ChatGPT, Perplexity, Claude | Oct 30, 2026 |
Why it matters
OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person requested the fetch. Anthropic's Claude-User is documented separately and does respect robots.txt. Blocking these agents is not a reliable privacy control and also stops the engines from reading your page live when users paste or ask about it.
How to do it
- Use authentication or noindex for content that must not be retrieved
- Leave user-triggered agents allowed for public marketing and documentation pages
- Check server logs for these user agents to see which pages people ask AI about
Sources
- Overview of OpenAI crawlers · OpenAI
- Perplexity crawlers · Perplexity
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic, Apr 7, 2026
Change history
| Date | Change | What changed |
|---|---|---|
| Oct 30, 2026 | Added | Added to the rulebook. |
Checked against its sources every Monday. Only content changes are listed.