Googlebot: what it is and how it crawls your site
Googlebot is Google's main web crawler for Google Search, and it also supports Google Images, Video, News and Discover. Its robots.txt token is Googlebot, and Google says it always obeys robots.txt for automatic crawls. Blocking it stops crawling of your pages and costs you Google Search visibility. Google renders pages with evergreen Chromium.
| Operator | Google, for Google Search |
|---|---|
| Purpose | Search engine |
| robots.txt token | Googlebot |
| User agent | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Follows robots.txt | Yes. Google's common crawlers, including Googlebot, always obey robots.txt rules when crawling automatically. |
| Runs JavaScript | Yes |
| IP ranges | Published list |
| Documentation |
What Googlebot does
Googlebot is Google's main crawler for Google Search. It fetches pages for the Search index, and the same crawl feeds Search features, Google Images, Video, News and Discover. Google renders pages with an evergreen version of headless Chromium.
Should you block Googlebot?
Allow Googlebot; blocking it costs you Google Search visibility, and robots.txt is not the way to deindex a page. Blocking Googlebot in robots.txt stops Google from crawling your pages, which stops them from being kept fresh in Google Search. To remove a page from the index, Google's guidance is noindex rather than robots.txt.
User-agent: Googlebot
Allow: /User-agent: Googlebot
Disallow: /Rules that apply
- Rule 06: Use noindex, not robots.txt Disallow, to keep a page out of search results
- Rule 27: Serve the same content, links and markup on mobile as on desktop
- Don't use crawl-delay to throttle Googlebot; Google ignores it
- Rule 31: Check that your site is included in the Search generative AI setting in Search Console
- Rule 03: Audit your CDN and WAF bot settings; they can block AI crawlers that robots.txt allows
Questions
What is the Googlebot user agent string?
Google documents a smartphone string ending in (compatible; Googlebot/2.1; +http://www.google.com/bot.html), with Chrome/W.X.Y.Z as a version placeholder, and a desktop variant. Use a wildcard for the Chrome version when searching logs.
Does Googlebot respect robots.txt?
Yes. Google states that its common crawlers, including Googlebot, always obey robots.txt rules for automatic crawls. Because user agent strings can be spoofed, Google advises verifying visitors, for example against its published IP ranges.
Does Googlebot render JavaScript?
Yes. Google says Search runs JavaScript with an evergreen version of Chromium. Pages with a 200 status are queued for rendering, which can take longer than a few seconds.
Should I block Googlebot?
Not if you want to appear in Google Search. To keep a page out of the index, Google's guidance is to use noindex rather than robots.txt.
Sources
- Google's common crawlers · Google
- Overview of Google crawlers and fetchers · Google
- Understand JavaScript SEO basics · Google
Checked against the operators' documentation every Monday. Last reviewed Nov 2, 2026.