Skip to content
GEO and AI search

Google-Extended (robots token)

Google-Extended is a robots.txt token that lets a site block its content from being used to train future Gemini and Vertex AI generative models, separately from Googlebot. Blocking it does not affect normal Google Search crawling, ranking, or a page's eligibility to appear in AI Overviews or AI Mode.

Also called: Google-Extended

Why it matters to a buyer

Google-Extended is one of the most commonly misunderstood tokens in this space: blocking it does not remove a site from Google AI Overviews or AI Mode, since those features are powered through ordinary Googlebot crawling and indexing, not through the training-data pipeline this token controls. A buyer who wants to stay out of AI Overviews cannot achieve that by disallowing Google-Extended, and a buyer who blocks it expecting no downside should understand it only affects whether that content helps train future Gemini and Vertex models.

How it is measured or applied

The token is added as its own line in robots.txt, disallowed or allowed independent of the main Googlebot rules, since Google-Extended has no separate HTTP user agent of its own and instead governs how content already fetched by Google's crawlers may be used for generative model training. A business decides its stance based on whether it wants its content used to train Google's future models, weighing that against the fact that blocking it has no effect on ordinary Search visibility. It is verified by checking the live robots.txt file directly rather than through any Search Console report.

Sources

  1. Google's common crawlers, Google Search Central

Thirty minutes to see where your site is invisible.

Book a free consultation with Ibrahem and Jeremy. We look at your site on search, AI answers and conversion while we talk, and you leave with a prioritised plan.

30 minutes. No pitch deck. You leave with a prioritised plan.