YouBot: what it is and how to manage it
YouBot is the web crawler of You.com, used to provide real-time search results to its users. Its robots.txt token is YouBot. You.com says it fully respects robots.txt and crawl-delay, with changes applying within about 30 minutes. Blocking it with Disallow: / keeps your pages out of You.com crawling. Its documentation does not mention AI model training.
| Operator | You.com, for You.com search |
|---|---|
| Purpose | AI search |
| robots.txt token | YouBot |
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36 |
| Follows robots.txt | Yes. You.com says YouBot fully respects robots.txt, including user-agent rules and crawl-delay, and caches the file for 30 minutes. |
| Runs JavaScript | Not documented |
| IP ranges | Not published |
| Documentation | You.com |
What YouBot does
YouBot crawls publicly accessible web pages to provide real-time search results for You.com users. You.com's documentation does not mention AI model training. It supports HTML, PDF, plain text, XML/RSS and structured data, and does not state whether it executes JavaScript.
Should you block YouBot?
Allow YouBot if you want your pages to appear in You.com results. Disallowing YouBot stops it from crawling your site, so your pages are not fetched for You.com search results. You.com says a robots.txt change can take up to 30 minutes to apply.
User-agent: YouBot
Allow: /User-agent: YouBot
Disallow: /Rules that apply
- Rule 01: Allow OAI-SearchBot, PerplexityBot and Claude-SearchBot in robots.txt if you want AI citations
- Rule 03: Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows
- Rule 04: Ship primary content in the initial HTML; most AI crawlers don't execute JavaScript
Questions
What is YouBot?
YouBot is You.com's web crawler. You.com says it crawls publicly accessible pages to provide comprehensive, real-time search results for its users.
Does YouBot respect robots.txt?
Yes. You.com says YouBot fully respects robots.txt directives, including user-agent-specific rules and crawl-delay. It caches robots.txt for 30 minutes, so changes can take up to 30 minutes to apply.
What is the YouBot user agent and IP range?
You.com documents the user agent as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36. It lists 68.67.112.0/24 and notes requests may use other addresses.
How do I block YouBot?
Add a robots.txt group with User-agent: YouBot and Disallow: / to block the whole site, or list specific paths. You.com also says returning 404 removes a page from its index, 429 reduces crawl rate and 503 pauses crawling.
Sources
- YouBot: You.com's Web Crawler · You.com
Checked against the operators' documentation every Monday. Last reviewed Nov 2, 2026.