Crawling & indexing rules
If a crawler cannot fetch and index a page, nothing else on this page matters. These rules decide whether Google, Bing and the AI engines can see you at all.
Reviewed Oct 30, 2026 · next review Nov 2, 2026
Impact split
- High impact 4
- Medium impact 6
- Low impact 1
Engines covered
Rules in this area that apply to each engine
- Google 7 of 11
- Bing 7 of 11
- ChatGPT 7 of 11
- Perplexity 4 of 11
- Gemini 2 of 11
- Claude 5 of 11
High impact 4
Allow OAI-SearchBot, PerplexityBot and Claude-SearchBot in robots.txt if you want AI citations
Each AI engine runs a dedicated search crawler that decides whether your pages can be shown and linked in answers.
Treat training bots and search bots as separate robots.txt decisions
GPTBot, ClaudeBot and the Google-Extended token control model training, while OAI-SearchBot, Claude-SearchBot and Googlebot control search visibility.
Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows
Since July 1, 2025 Cloudflare blocks known AI crawlers by default on new domains, and other WAFs ship similar bot rules.
Ship primary content in the initial HTML; most AI crawlers don't execute JavaScript
Google renders JavaScript but queues it, and still recommends server-side or pre-rendering.
Medium impact 6
Don't use Google-Extended to control Search; it only governs Gemini training and grounding
Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
Use noindex, not robots.txt Disallow, to keep a page out of search results
A Disallow stops crawling, but Google can still index the URL from links and show it without a snippet.
Submit an XML sitemap with lastmod values that change only on significant updates
Google uses lastmod only when it is consistently and verifiably accurate, and a copyright year change does not count.
Ping IndexNow when URLs are added, changed or deleted, for Bing and partner engines
IndexNow lets you push changed URLs instead of waiting for a crawl, and a submission is shared with every participating engine.
Point redirects, rel=canonical, sitemaps and internal links at the same canonical URL
Google treats redirects and rel=canonical as strong signals and sitemap inclusion as a weak one, and the signals stack.
Return real 404/410 for missing pages and keep critical HTML under Googlebot's 2 MB fetch limit
Google asks for meaningful status codes so it knows whether a URL can be indexed, and soft 404s waste crawl.
Low impact 1
Don't rely on robots.txt to stop ChatGPT-User or Perplexity-User; they fetch on a user's behalf
OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person requested the fetch.
No longer works 2
Stop pinging Google's sitemap endpoint and setting priority or changefreq
Do insteadSubmit sitemaps in Search Console or via a Sitemap: line in robots.txt
Don't use crawl-delay to throttle Googlebot; Google ignores it
Do insteadLet Google adjust crawl rate automatically, or return 503/429 temporarily under overload