Google-Extended explained
Google-Extended is a robots.txt token that controls Gemini training, not a crawler. Blocking it does not remove you from Search or AI Overviews.
Google-Extended is a robots.txt token, introduced in September 2023, that lets a publisher control whether content Google has already crawled may be used to train future Gemini models and to ground Gemini's answers. It is not a crawler, has no separate user agent, and disallowing it does not affect Google Search rankings or AI Overviews eligibility.
That last sentence is worth repeating because it is the most common misunderstanding in AI search optimization: teams disallow Google-Extended expecting it to remove them from Google's AI features, and nothing about their visibility changes, because that was never what the token controlled.
What does the Google-Extended token actually control?
Training data and grounding for Gemini, specifically. Google's own documentation states that crawling is done with existing Google user agent strings and that the Google-Extended token is used "in a control capacity" rather than as a fetcher of its own — it is evaluated against content Googlebot already retrieved, not fetched separately [1].
Sept 28, 2023
That framing — a control rather than a crawler — is what makes Google-Extended different from every AI bot discussed elsewhere in this cluster, including how Perplexity ranks its sources and how Claude chooses its sources, both of which involve a real, separately blockable fetcher.
It also means the usual verification trick — fetch a page under a crawler's user agent and confirm the response — does not apply here the way it does for those other bots. There is nothing to fetch under, because Google-Extended never appears in a server log as a request. The only way to check the setting is to read your own robots.txt file, not to watch traffic.
This distinction confuses teams who set up crawler rules by copying a checklist written for training bots in general. A rule that correctly blocks GPTBot or ClaudeBot at the network edge has nothing to block for Google-Extended, because blocking happens by Google's own systems honouring the robots.txt line, not by a firewall intercepting a request that never comes.
Does it affect Google Search or AI Overviews at all?
No, on both counts, by Google's own statement: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" [1]. AI Overviews and AI Mode are Google Search features and depend on the same Googlebot crawl and index that ordinary results do, not on Google-Extended [3].
| Outcome | What actually governs it | Does Google-Extended affect it? |
|---|---|---|
| Inclusion in Google Search | Googlebot's crawl and index | No |
| Ranking position | Google's normal ranking systems | No |
| Appearing in AI Overviews / AI Mode | Googlebot's crawl and index, same as Search | No |
| Training future Gemini models | The Google-Extended token | Yes |
| Grounding Gemini app answers in your content | The Google-Extended token | Yes |
How do you actually set it, and should you?
Add the token to robots.txt
A `User-agent: Google-Extended` block with a `Disallow: /` line opts your whole site out of training and grounding use. Scope it to a path with a narrower `Disallow` if you only want to exclude part of the site.
Decide training and citation separately
Disallowing Google-Extended does not touch Googlebot's own crawl permissions, so it has no bearing on whether AI Overviews can still cite you — that stays governed by your ordinary robots.txt rules for Googlebot.
Do not treat it as a privacy or competitive fix by itself
It only covers Google's own Gemini training and grounding. It has no effect on OpenAI, Anthropic, Perplexity or any other company's crawlers, each of which has its own separate opt-out.
How is this different from OpenAI's or Anthropic's approach?
Most other assistants separate their crawlers by function the way Google separates its controls, but with real, independently blockable fetchers behind each one. ChatGPT and Perplexity handle this differently again, each publishing its own named crawlers for training, live retrieval and search indexing, and each of those can be allowed or blocked on its own regardless of what you decide about Google-Extended.
The pattern worth remembering across all of them: a training opt-out and a citation opt-in are two different decisions, made through two different controls, at every one of these companies. Google-Extended is simply the clearest single example, because Google has stated the boundary explicitly rather than leaving it to be inferred. Our AI Overviews and AI Mode service starts by confirming which of these settings a site actually has right, rather than assuming the team's mental model of any one token is correct.
A related question we get from clients migrating a robots.txt file between sites: does removing a Google-Extended line reset the decision to "allow training" by default? Yes — the absence of the token is treated as no restriction, the same way a missing Disallow line for any crawler defaults to permission rather than denial. If the previous site intentionally opted out, that line has to be carried over explicitly, not assumed to persist through a migration.
The broader lesson generalises past this one token: read what a control actually says it governs before writing it into a checklist, because the name alone — "Extended," "training," "bot" — invites assumptions that Google's own documentation does not support. The definition in the primary source is the whole answer, and it rarely matches the version that circulates as received wisdom.
Questions people also ask
Is Google-Extended a separate crawler I can block at my firewall?
No. It has no HTTP user agent of its own. Google crawls with the same Googlebot request it always used, and the Google-Extended line in robots.txt is read afterward as a control over how that already-crawled content may be used.
Does disallowing Google-Extended hurt my rankings?
No, by Google's own documentation. It does not affect inclusion in Google Search and is not used as a ranking signal. It only governs training and grounding for Gemini apps and Vertex AI.
Does Google-Extended control AI Overviews or AI Mode eligibility?
No. Appearing in AI Overviews and AI Mode depends on being crawled and indexed by Googlebot, the same as normal Search. Google-Extended governs training use, a separate decision from whether Googlebot can crawl you at all.
Should every business disallow Google-Extended?
Not automatically. It costs nothing to disallow if you have no reason to help train Gemini, but it is a training decision, not a visibility one, so it should not be conflated with anything that affects whether you show up in search or AI answers.
Sources
- Overview of Google crawlers and fetchers — Google Search Central (checked 23 September 2026)
- An update on web publisher controls — Google (checked 23 September 2026)
- AI features in Search — Google Search Central (checked 23 September 2026)
Find out which AI training and citation controls your site actually has set
Book a free 30-minute consultation with Ibrahem and Jeremy. We look at your site while we talk, and you leave with a prioritised plan.
Who wrote this
Ibrahem HelalCo-founder
Leads strategy and delivery.
Works on: SEO, GEO
Reviewed before publishing by Jeremy Zava, Co-founder.
- AI search optimization: the 2026 guideHow to get cited by ChatGPT, Gemini, Perplexity, Claude, Copilot and Google's AI features: crawler access, citable pages, third-party proof and measurement.
- How Perplexity picks and ranks its sourcesPerplexity crawls its own index, fetches pages live, and cites more sources per answer than any rival — with the closest overlap to Google's rankings.
- How Claude chooses which sources to citeClaude searches Anthropic's own index, cites the fewest sources of any major assistant, and often names a brand without linking it at all.
- ChatGPT vs Perplexity: which cites more?ChatGPT cites about 4.5 sources per answer; Perplexity cites about 19. What that gap means for which pages get picked, and how to win a citation in each.