Google · Training
Google-Extended
Every other entry in this directory describes a program that makes requests. This one describes a permission. Google calls it a standalone product token publishers use to manage whether content Google has crawled may be used for training future Gemini models and for grounding: pulling text from the Search index and handing it to a model at prompt time. Disallowing it does not reduce crawling by a single request; Googlebot still fetches your pages on exactly the same schedule. It changes what Google is permitted to do with what it already holds.
- Operated by
- Purpose
- Training
- robots.txt token
- Google-Extended
- Verification
- Not a crawler
Collects pages into a corpus used to train models.
Can Google-Extended be verified?
Google-Extended is a robots.txt permission token, not a crawler. No request ever arrives carrying it as a user agent, so there is nothing to verify and nothing to see in traffic logs. It exists so you can state a preference that Google reads separately, using the crawlers it does operate.
This token issues no requests, so there is nothing to confirm. This entry was last reviewed against Google's own documentation on .
User agent
None. Google-Extended is only ever written into a robots.txt file. It never appears in a request header, so searching your access logs for it will always come back empty.
robots.txt for Google-Extended
It IS the robots.txt mechanism. There is no crawler here to obey or disobey anything: Google states the token has no separate HTTP user-agent string, that crawling is done with existing Google user agents, and that the robots.txt token is used purely in a control capacity.
Block
User-agent: Google-Extended
Disallow: /Allow
User-agent: Google-Extended
Allow: /robots.txt is a request, not an enforcement mechanism. It is honoured by convention, and a crawler that ignores it is stopped at your edge, not in a text file.
What blocking Google-Extended costs you
The trade is unusually clean, and Google states the safe half in writing: this token does not affect a site's inclusion in Search and is not used as a ranking signal. You keep your rankings. What you give up is your content's availability for grounding answers in Gemini apps and in Grounding with Google Search on Vertex AI, so if Gemini currently summarises or cites your material, that stops: a real loss of an emerging referral surface for publishers whose value is being the authority a model reaches for. Note the boundary: disallowing this does NOT remove you from AI Overviews or AI Mode, which are governed by Googlebot.