CCBot: what it is and how to block it
CCBot is the web crawler of Common Crawl, a non-profit that publishes an open repository of web crawl data free to researchers, companies and individuals. Its robots.txt token is CCBot, and User-agent: CCBot with Disallow: / blocks it. Common Crawl says CCBot obeys robots.txt and Crawl-delay and does not execute JavaScript.
| Operator | Common Crawl, for Common Crawl open web dataset |
|---|---|
| Purpose | AI training |
| robots.txt token | CCBot |
| User agent | CCBot/2.0 (https://commoncrawl.org/faq/) |
| Follows robots.txt | Yes. Common Crawl says CCBot checks robots.txt first, rechecks it periodically, obeys Crawl-delay and honors nofollow on links. |
| Runs JavaScript | No |
| IP ranges | Published list |
| Documentation | Common Crawl |
What CCBot does
CCBot is the crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data. The data is available at no cost to researchers, companies and individuals. Common Crawl's pages do not mention AI training specifically, but the dataset is public and open to any user. CCBot does not execute JavaScript or use cookies.
Should you block CCBot?
Allow or block CCBot as a policy choice about open datasets, and verify it by reverse DNS because other crawlers falsely claim the name. Disallowing CCBot stops Common Crawl from fetching your pages for future crawls. Common Crawl's pages do not describe other effects on search or AI products.
User-agent: CCBot
Disallow: /User-agent: CCBot
Allow: /Rules that apply
- Rule 02: Treat training bots and search bots as separate robots.txt decisions
- Rule 04: Ship primary content in the initial HTML; most AI crawlers don't execute JavaScript
- Rule 03: Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows
Questions
What is CCBot?
CCBot is the crawler operated by Common Crawl, a non-profit that maintains an open repository of web crawl data. The data is made available at no cost for research and other uses.
Does CCBot respect robots.txt?
Yes. Common Crawl says CCBot checks robots.txt first, periodically rechecks it, obeys Crawl-delay and honors nofollow. It also slows down automatically when a server returns HTTP 429 or 5xx errors.
How do I block CCBot?
Add a robots.txt group with User-agent: CCBot and Disallow: /. Common Crawl also warns that some crawlers falsely claim to be CCBot, and publishes its IP ranges as JSON for verification.
What is the CCBot user agent?
Common Crawl documents it as CCBot/2.0 (https://commoncrawl.org/faq/). Its dedicated IP ranges are published at index.commoncrawl.org/ccbot.json, and IPv4 addresses support reverse DNS.
Sources
- CCBot · Common Crawl
- Frequently Asked Questions · Common Crawl
Checked against the operators' documentation every Monday. Last reviewed Nov 2, 2026.