Try free

Cloudflare AI crawl control: stop your CDN from blocking AI search bots

If your site is behind Cloudflare, check its AI crawler settings before you trust robots.txt. Since July 1, 2025 Cloudflare blocks known AI crawlers by default on new domains, and a firewall block overrides a permissive robots.txt. Allow the search bots you want citations from, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, even if you keep training bots blocked.

The rule: Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows

Impact Evidence Applies to Last checked
High Official guidance Bing, ChatGPT, Perplexity, Claude Oct 30, 2026

Why it matters

Since July 1, 2025 Cloudflare blocks known AI crawlers by default on new domains, and other WAFs ship similar bot rules. A firewall block overrides a permissive robots.txt, so the crawler never sees your page. OpenAI also asks sites to allow requests from its published IP ranges for OAI-SearchBot.

How to do it

  1. Open your CDN bot management or AI crawler settings and list which AI agents are blocked or challenged
  2. Allow the search bots you want citations from, even if you block training bots
  3. Allowlist OpenAI's published OAI-SearchBot IP ranges if your WAF challenges unknown bots
  4. Verify with server logs that the bots receive 200 responses, not 403 or challenge pages

Questions

Does Cloudflare block AI crawlers by default?

Yes. Since July 1, 2025 Cloudflare blocks known AI crawlers by default on new domains. Site owners can change the default and allow or block individual AI bots case by case.

Can I block AI training bots but allow AI search bots?

Yes. Cloudflare lets owners permit or deny crawling per stage of the AI lifecycle: training, fine-tuning and inference, and allow specific verified crawlers. Allow search bots like OAI-SearchBot and PerplexityBot, and block training bots like GPTBot if you prefer.

Why is OAI-SearchBot blocked when robots.txt allows it?

A CDN or WAF rule is challenging or blocking it before robots.txt matters. Allowlist OpenAI's published OAI-SearchBot IP ranges at openai.com/searchbot.json, then check server logs to confirm the bot receives 200 responses rather than 403s or challenge pages.

What is Cloudflare pay per crawl?

A Cloudflare service that lets site owners set a rate AI bots must pay to crawl their site, with the owner compensated for each crawl. It is a choice for content you want to license, not a substitute for allowing the search bots you want citations from.

Sources

Change history

Date Change What changed
Oct 30, 2026 Added Added to the rulebook.

Checked against its sources every Monday. Only content changes are listed.

Related rules