AI crawler list: every AI bot and its user agent

19 crawlers from 12 companies fetch web pages for AI products today. They do different jobs: search engine and AI search crawlers decide whether you can appear and be cited in answers, user-triggered fetchers read a page when someone asks about it, and 8 training crawlers collect content for model training. Blocking a training crawler opts you out of training without removing you from search or AI answers.

Checked against each operator's documentation every Monday. Last reviewed Nov 2, 2026.

Search engine crawlers

Classic web search crawlers. Their indexes also feed AI answers such as Google AI Overviews and Microsoft Copilot, so blocking them removes you from both.

Crawler	Operator	robots.txt token	Follows robots.txt
Applebot	Apple	`Applebot`	Yes
Bingbot	Microsoft	`bingbot`	Yes
Googlebot	Google	`Googlebot`	Yes

AI search crawlers

Build the indexes AI assistants search and cite. Block one and that assistant stops linking to you.

Crawler	Operator	robots.txt token	Follows robots.txt
Claude-SearchBot	Anthropic	`Claude-SearchBot`	Yes
DuckAssistBot	DuckDuckGo	`DuckAssistBot`	Yes
OAI-SearchBot	OpenAI	`OAI-SearchBot`	Yes
PerplexityBot	Perplexity	`PerplexityBot`	Yes
YouBot	You.com	`YouBot`	Yes

User-triggered fetchers

Fetch a page live because a person asked an assistant about it. Several of them do not follow robots.txt rules.

Crawler	Operator	robots.txt token	Follows robots.txt
ChatGPT-User	OpenAI	`ChatGPT-User`	Partly
Claude-User	Anthropic	`Claude-User`	Yes
Perplexity-User	Perplexity	`Perplexity-User`	No

AI training crawlers and tokens

Collect content to train AI models. Blocking them is an opt-out from training; it does not remove you from search or AI answers.

Crawler	Operator	robots.txt token	Follows robots.txt
Amazonbot	Amazon	`Amazonbot`	Yes
Applebot-Extended	Apple	`Applebot-Extended`	Yes
Bytespider	ByteDance	`Bytespider`	Not documented
CCBot	Common Crawl	`CCBot`	Yes
ClaudeBot	Anthropic	`ClaudeBot`	Yes
Google-Extended	Google	`Google-Extended`	Yes
GPTBot	OpenAI	`GPTBot`	Yes
Meta-ExternalAgent	Meta	`meta-externalagent`	Yes

robots.txt templates

Copy the template that matches your decision into the robots.txt of every host that serves content. Both keep Googlebot, Bingbot and Applebot untouched, so classic search is not affected.

Opt out of AI training, stay in AI search and answers

# Opt out of AI training, stay in AI search and answers

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: YouBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

Block every AI crawler, keep classic search engines

# Block AI crawlers, keep classic search engines

User-agent: Claude-SearchBot
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

Blocking AI search crawlers removes you from those assistants' answers. Read why training and search bots are separate decisions before you use the second template.

Questions

Which AI crawlers should I block?

Only training crawlers, and only if you want to opt out of model training. Search and user-triggered crawlers decide whether AI assistants can find, read and cite your pages, so blocking them costs visibility. Treat training and search as separate robots.txt decisions.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects data for training OpenAI models. ChatGPT search relies on OAI-SearchBot, and OpenAI states that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. Block GPTBot and allow OAI-SearchBot to opt out of training and stay citable.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended only controls whether your content is used to train Gemini models and for Gemini grounding. AI Overviews and AI Mode are governed by Googlebot access, snippet controls and the Search generative AI setting in Search Console.

Do AI crawlers follow robots.txt?

Most documented training and search crawlers do, though Bytespider, which ByteDance does not document, has been reported to ignore it. User-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it, because a person requested the page. A firewall or CDN rule can also block crawlers that robots.txt allows.

Can AI crawlers read JavaScript?

Most cannot. An analysis of crawler traffic by Vercel found that GPTBot, OAI-SearchBot, ChatGPT-User and ClaudeBot fetch JavaScript files but do not execute them. Put your main content in the initial HTML so these crawlers can read it.