GPTBot: what it is and how to block or allow it
GPTBot is OpenAI's web crawler that collects content which may be used to train its generative AI foundation models. Its robots.txt token is GPTBot. Disallowing it opts your site out of training use, and OpenAI says it does not affect OAI-SearchBot, the separate crawler that decides whether your pages appear in ChatGPT search results.
| Operator | OpenAI, for ChatGPT and OpenAI models |
|---|---|
| Purpose | AI training |
| robots.txt token | GPTBot |
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot |
| Follows robots.txt | Yes. Disallowing GPTBot in robots.txt signals that your content should not be used to train OpenAI's generative AI foundation models. |
| Runs JavaScript | No |
| IP ranges | Published list |
| Documentation | OpenAI |
What GPTBot does
GPTBot is OpenAI's crawler for collecting web content that may be used to train its generative AI foundation models. It is separate from OAI-SearchBot and ChatGPT-User, and each has its own robots.txt setting.
Should you block GPTBot?
Treat GPTBot as a training choice, and block or allow it separately from OAI-SearchBot, which controls ChatGPT search visibility. Blocking GPTBot tells OpenAI not to use your content for training. OpenAI states that this does not affect OAI-SearchBot, since each setting is independent.
User-agent: GPTBot
Disallow: /User-agent: GPTBot
Allow: /Rules that apply
- Rule 02: Treat training bots and search bots as separate robots.txt decisions
- Rule 03: Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows
- Rule 04: Ship primary content in the initial HTML; most AI crawlers don't execute JavaScript
Questions
Should I block GPTBot?
That depends on whether you want your content used for model training. OpenAI documents GPTBot as a training crawler only, so blocking it is a policy choice rather than a visibility decision.
Does blocking GPTBot remove me from ChatGPT?
OpenAI states that blocking GPTBot does not affect OAI-SearchBot, because each robots.txt setting is independent. ChatGPT search inclusion is controlled by OAI-SearchBot, not GPTBot.
What is the GPTBot user agent string?
OpenAI documents it as: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. The version number may change, so match on the GPTBot token.
Does GPTBot respect robots.txt?
Yes. OpenAI documents that disallowing GPTBot signals your content should not be used for training. OpenAI also publishes GPTBot's IP ranges at openai.com/gptbot.json so you can verify requests.
Sources
- Overview of OpenAI Crawlers · OpenAI
- The rise of the AI crawler · Vercel, Dec 17, 2024
Checked against the operators' documentation every Monday. Last reviewed Nov 2, 2026.