Try free

Allow OAI-SearchBot, PerplexityBot and Claude-SearchBot in robots.txt

OAI-SearchBot is the crawler OpenAI uses to surface websites in ChatGPT search, and OpenAI says sites that opt out of it will not be shown in ChatGPT search answers. Allow it, PerplexityBot and Claude-SearchBot explicitly in robots.txt on every host that serves content. OpenAI and Perplexity say changes take up to about 24 hours to apply.

The rule: Allow OAI-SearchBot, PerplexityBot and Claude-SearchBot in robots.txt if you want AI citations

Impact Evidence Applies to Last checked
High Official guidance ChatGPT, Perplexity, Claude Oct 30, 2026

Why it matters

Each AI engine runs a dedicated search crawler that decides whether your pages can be shown and linked in answers. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and Perplexity and Anthropic say blocking their search bots reduces visibility. Blocking these bots removes you from the answer pool regardless of content quality.

How to do it

  1. Open /robots.txt and check for Disallow rules under OAI-SearchBot, PerplexityBot, Claude-SearchBot or a catch-all User-agent: *
  2. Add explicit Allow groups for each search bot you want to be cited by
  3. Repeat for every subdomain that serves content, since robots.txt is per host
  4. Expect about 24 hours before OpenAI and Perplexity pick up robots.txt changes

Example

robots.txt
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Questions

What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content that may be used to train OpenAI's foundation models, and disallowing it signals you don't want that. OAI-SearchBot surfaces sites in ChatGPT search. You can block GPTBot and still allow OAI-SearchBot, which keeps you eligible for ChatGPT search citations.

Should I block PerplexityBot?

Not if you want Perplexity citations. PerplexityBot surfaces and links websites in Perplexity search results, and Perplexity says it is not used to crawl content for AI foundation models. Allow it in robots.txt and permit its published IP ranges.

What is Claude-SearchBot?

Claude-SearchBot is Anthropic's crawler for improving search results in Claude. Anthropic says blocking it prevents your content from being indexed for search and may reduce your visibility. It is separate from ClaudeBot, which collects training data, and Claude-User, which fetches pages when a person asks Claude a question.

Does ChatGPT-User respect robots.txt?

Not necessarily. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiates the fetch, and it is not used to decide Search eligibility. Perplexity-User likewise generally ignores robots.txt. OAI-SearchBot and PerplexityBot are the bots that robots.txt controls for search.

Sources

Change history

Date Change What changed
Oct 30, 2026 Added Added to the rulebook.

Checked against its sources every Monday. Only content changes are listed.

Related rules