Google · Search index
Googlebot
The oldest and broadest thing in this directory, and the reason the robots.txt convention exists at all. Google's scope statement is deliberately wide: preferences addressed to this token affect Google Search including Discover and all Search features, plus Images, Video and News. That phrase *all Search features* does quiet but enormous work in 2026, because AI Overviews and AI Mode are Search features: Google's own AI guidance names robots.txt directives for Googlebot as the control for how sites are crawled for Search. There is no separate AI-Overviews crawler to allow or deny; the index this builds is the substrate.
- Operated by
- Purpose
- Search index
- robots.txt token
- Googlebot
- Verification
- IP verified
Crawls continuously to build and refresh an AI search index.
How to verify Googlebot
Google publishes the IP ranges Googlebot crawls from, and we fetch that list on a schedule. A request claiming to be Googlebot can therefore be checked against 2,163 published ranges: one that does not match is not this crawler.
Published ranges last confirmed by us on .
Google documents reverse-DNS verification: a genuine request resolves to a hostname under .googlebot.com or .geo.googlebot.com that forward-resolves back to the same address.
User agent
Google publishes this user agent for Googlebot. Match on the Googlebot product token rather than the whole string: vendors revise the surrounding version and URL fragments without notice.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36robots.txt for Googlebot
Yes, and this is the class Google is most explicit about: common crawlers always respect robots.txt rules for automatic crawls. The token also acts as the fallback group for several sub-crawlers, including Googlebot-Image, Googlebot-Video, Googlebot-News and Google-CloudVertexBot, when no more specific group matches.
Block
User-agent: Googlebot
Disallow: /Allow
User-agent: Googlebot
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking Googlebot costs you
The maximum-cost block available to a website operator, and it is not confined to blue links. A disallow removes you from organic search, Images, Video, News and Discover at once, and because AI Overviews and AI Mode draw on the Search index rather than a separate corpus, it removes you from those too, Google having stated there are no additional requirements to appear in them. Operators who want to stay indexed but limit how much text Google surfaces should reach for `nosnippet`, `data-nosnippet`, `max-snippet` or `noindex`, which is the granular control Google points to. A blanket disallow here is the sledgehammer.