More
Developers Community Contact Sign inClaude searches the web through Claude-SearchBot and cites what it finds. These rules decide whether Claude can read and cite you.
Reviewed Oct 30, 2026 · next review Nov 2, 2026
Share of each area that applies to Claude
Each AI engine runs a dedicated search crawler that decides whether your pages can be shown and linked in answers.
GPTBot, ClaudeBot and the Google-Extended token control model training, while OAI-SearchBot, Claude-SearchBot and Googlebot control search visibility.
Since July 1, 2025 Cloudflare blocks known AI crawlers by default on new domains, and other WAFs ship similar bot rules.
Google renders JavaScript but queues it, and still recommends server-side or pre-rendering.
OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person requested the fetch.
Google's helpful content guidance and its 2026 generative AI guide both say the same thing: content created for people with first-hand experience wins, and recycled summaries do not.
Google generally only follows <a> elements with an href, and links built on javascript: URLs or click handlers may not be extracted.
Google's 2026 AI guide recommends clear paragraphs, sections and headings for readers, and lists headings as a title link source.
Bing's February 2026 guidelines say single-topic pages are more likely to be selected as grounding sources and recommend placing essential information near the top and stating facts directly so they can be verified.
Google's 2026 AI guide tells site owners to offer a unique point of view and not recycle what others have said, and it describes AI answers as grounded in retrieved pages.
The Princeton GEO paper (KDD 2024) found that adding statistics, quotations and source citations raised visibility in generative engine responses by up to 40% on its benchmark, while keyword stuffing did not help.
Google's Search generative AI performance reports (June 2026, global since August 31, 2026) show impressions from AI Overviews and AI Mode, and Bing's AI Performance report (February 2026) shows Copilot citations and grounding queries.
Bing's 2026 guidelines ask for clear, consistent entity names for grounding.
Do insteadKeep an llms.txt only if you have docs that coding agents read