Bytespider: what it is and whether to block it
Bytespider is a web crawler attributed to ByteDance, the owner of TikTok. ByteDance publishes no official documentation for it. Known Agents describes it as allegedly collecting training data for ByteDance's language models. The robots.txt token is Bytespider. Whether it honors robots.txt is disputed: Fortune reported that it ignores it, while Known Agents measured high compliance.
| Operator | ByteDance, for ByteDance AI products (reported) |
|---|---|
| Purpose | AI training |
| robots.txt token | Bytespider |
| User agent | No full string documented. Match on the robots.txt token. |
| Follows robots.txt | Not documented. ByteDance publishes no crawler documentation, and third parties disagree: Fortune (Oct 2024) reported, citing Kasada research, that it ignores robots.txt, while Known Agents lists it as expected to follow robots.txt with a 95.7% measured compliance rate. |
| Runs JavaScript | Not documented |
| IP ranges | Not published |
| Documentation | Known Agents (Dark Visitors) |
What Bytespider does
Bytespider is a crawler attributed to ByteDance, the owner of TikTok. ByteDance has not published an official page describing it. Known Agents describes it as downloading web content that is allegedly used as training data for ByteDance's large language models, including those behind Doubao.
Should you block Bytespider?
Block Bytespider by token if you want out of ByteDance training data, and enforce it at the CDN or firewall because compliance is disputed. No official ByteDance documentation states what blocking does. Known Agents says the crawler is allegedly used for AI training data, so a Disallow would be an opt-out of that collection, if the bot honors it.
User-agent: Bytespider
Disallow: /User-agent: Bytespider
Allow: /robots.txt is not a reliable control for Bytespider. To keep content away from it, put the content behind a login or block the requests at your server or CDN.
Rules that apply
- Rule 02: Treat training bots and search bots as separate robots.txt decisions
- Rule 03: Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows
Questions
What is Bytespider?
Bytespider is a crawler attributed to ByteDance, the owner of TikTok. Fortune reported in October 2024, citing Kasada research, that ByteDance released it in April of that year. ByteDance has not published an official page describing it.
Does Bytespider respect robots.txt?
It is disputed. Fortune reported, citing Kasada research, that Bytespider does not respect robots.txt. Known Agents lists it as expected to follow robots.txt and reports a 95.7% measured compliance rate from its own data. ByteDance has not documented its behavior.
How do I block Bytespider?
Add a robots.txt group with User-agent: Bytespider and Disallow: /. Because compliance is disputed and any client can claim the name, enforce the block at the CDN or firewall if it matters to you.
What is the Bytespider user agent?
ByteDance does not document one. Known Agents shows an example string containing the Bytespider token and https://zhanzhang.toutiao.com/, and warns that any client can claim the identity, so a log match is only a clue.
Sources
- Bytespider · Known Agents (Dark Visitors)
- TikTok's parent launched a web scraper that's gobbling up the world's online data 25 times faster than OpenAI · Fortune, Oct 3, 2024
Checked against the operators' documentation every Monday. Last reviewed Nov 2, 2026.