Robots.txt Rules for AI Crawlers
A working robots.txt for AI crawlers names GPTBot, ClaudeBot and PerplexityBot explicitly and never blocks the ones your citations depend on.
Allow the crawlers you want and name them explicitly: don't rely on a bare wildcard. List OpenAI's, Anthropic's and Perplexity's bots by name, allow Google-Extended and Googlebot, and check your CDN and firewall separately, because they block AI user agents by default and robots.txt cannot override that.
This is one line item on the wider technical SEO checklist: a crawler rule that looks correct in the file but is overridden somewhere else in the stack.
On this page
Which AI crawlers actually need a line in your file?
More than most sites assume. Each major lab runs at least two bots with different jobs, and treating "AI crawler" as one thing is how a site accidentally blocks the one that mattered.
OpenAI publishes three: GPTBot trains future models, OAI-SearchBot builds the index behind ChatGPT's cited answers, and ChatGPT-User fetches a page live while someone is in a conversation [1]. Anthropic splits the same way: ClaudeBot trains, Claude-SearchBot indexes for citations, and Claude-User fetches live [2]. Perplexity runs PerplexityBot for its own crawl and index [3]. Google's case is different — Googlebot alone drives Search, AI Overviews and AI Mode, while Google-Extended is a separate opt-out that only affects Gemini app and Vertex AI training [4].
| Crawler | What it does |
|---|---|
| GPTBot (OpenAI) | Trains future models; blocking it only removes you from training data |
| OAI-SearchBot (OpenAI) | Builds the index behind ChatGPT search citations |
| ChatGPT-User (OpenAI) | Fetches a page live during an active conversation |
| ClaudeBot (Anthropic) | Trains future models |
| Claude-SearchBot / Claude-User (Anthropic) | Indexes for citations, and fetches live during a chat |
| PerplexityBot (Perplexity) | Crawls and indexes for Perplexity's own answers |
| Google-Extended (Google) | Gemini app and Vertex AI training only, not Search or AI Overviews |
Treat that table as your allow list. If a name is missing from your robots.txt, that job is unblocked by default, which is fine — but you should know that, rather than discover it later.
User-agent strings are matched literally, so spelling and case matter more than people expect. Gptbot instead of GPTBot will not match on most implementations, and a rule copied from an old blog post can quietly reference a bot name a lab has since renamed or split into two. Pull the exact strings from each lab's own documentation rather than from a secondhand list, and re-check them whenever a lab announces a new crawler — labs have split a single bot into a training and a retrieval version before, and a rule written for the old single name stops covering half the job.
What does a correct rule actually look like?
Explicit, one block per crawler, with a wildcard Allow: / underneath as a backstop.
Name each crawler in its own User-agent block
A shared wildcard already covers unnamed bots, but an explicit block survives someone later adding a narrower Disallow rule aimed at a different crawler that would otherwise sweep this one up by accident.
Allow the path, not just the crawler
Write `Allow: /` under each User-agent line. A crawler with no matching Allow rule falls back to whatever the general rule says, and a stray Disallow higher in the file can silently apply to it.
Do not duplicate Disallow rules per bot
Keep one set of Disallow paths that applies to everyone — admin routes, search results, staging — rather than maintaining it separately under each crawler, which is how rules drift out of sync.
Comment the file for the next person
A one-line comment above each crawler naming who it belongs to and what it is for saves the next engineer from disallowing GPTBot by mistake while tidying up an unrelated rule.
A minimal, explicit version names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Google-Extended, each with its own Allow: /, above your general rules. It is a handful of lines, and it is the part of this file most likely to be right once and then forgotten.
If your marketing site runs on a subdomain, check that subdomain's own robots.txt separately. Each host serves its own file, so a correct set of rules on the root domain says nothing about a docs or blog subdomain running on different infrastructure, and it is common for the second one to still carry whatever generic template shipped with the platform.
Should you block the training crawlers to keep content out of models?
That is a real, defensible choice — but be clear about what it does and does not buy you.
Most businesses want the opposite: to be found and cited. If that is you, allow the training bots too, because being present in a model's parametric memory is part of how it recognizes your brand as a real answer later, not just a retrieved one.
Does blocking Google-Extended hurt your AI Overviews traffic?
No, and this is the single most common mistake in this whole area. Google-Extended is a training-only signal for the Gemini app and Vertex AI [4]. Search itself — organic results, AI Overviews, AI Mode — runs entirely on Googlebot, and there is no separate opt-out for any of it.
So a site that blocks Google-Extended to "stay out of AI" has not touched its AI Overviews visibility at all; it has only opted out of a training use case most sites never think about either way. If you are trying to reduce your footprint in Google's generative features specifically, robots.txt is not where that decision lives.
Where do blocks actually hide, if the file itself looks fine?
Off the file entirely, more often than people expect. CDNs, hosting platforms and web application firewalls ship bot-management features, and several of them classify AI crawlers as bots to challenge or block by default — independent of whatever your robots.txt says.
The failure mode is quiet. There is no error, no blocked-request log entry a normal person checks, and no drop that shows up anywhere except a citation rate that never grows. A site can carry a perfect, explicit robots.txt file and still be invisible to every AI crawler because a setting three layers below it disagrees.
How do you confirm a crawler can actually reach a page?
By asking it directly rather than assuming the configuration is correct.
Fetch the page with the crawler's user agent
Use curl with the exact user-agent string from the lab's published list and confirm you get a 200 response and full HTML, not a 403 or a CAPTCHA page.
Fetch it with no JavaScript at all
Some retrieval crawlers execute little or no client-side script. If the page's key content only appears after a script runs, a crawler that does not run it sees a near-empty page even with a 200 status.
Check your access logs for the crawler by name
A crawler that is truly allowed shows up in your logs on a regular cadence. One that is silently blocked at the edge often never appears at all, which is the tell that something upstream is intercepting it before your server logs the request.
Re-check after any platform or firewall change
A CDN migration, a new WAF ruleset or a hosting provider switch is exactly when these blocks get re-introduced, usually by a default nobody reviewed.
This is the same discipline as fixing any other indexing problem: don't assume a fix worked because the change looked right in a file. Confirm the crawler's own view of the page.
Who owns this file, and how often should it be rechecked?
Nobody, on most teams — which is exactly the problem. A robots.txt file is usually written once during a launch or a migration and then left alone, while the crawlers it names keep changing. A lab can add a new bot, split an existing one into a training and a retrieval version, or rename one entirely, and a file that was correct on the day it shipped goes quietly out of date with nothing in your monitoring to flag it.
Robots.txt is one of the few technical SEO files that never throws an error when it's wrong. A typo blocks a crawler silently and forever, until someone happens to check, so I'd rather put a recurring calendar reminder on this than trust that it stays right on its own.
A practical cadence: recheck the file quarterly against each lab's current published list, and immediately after any CDN, hosting or firewall change, since that is when edge-level blocks tend to get reintroduced. Assign the check to whoever owns technical SEO on your team rather than leaving it to whoever happens to remember, because a file nobody owns is a file nobody rechecks.
What should never sit on a page you want quoted?
A directive that forbids the excerpt you are trying to earn. A nosnippet meta tag, an X-Robots-Tag header with nosnippet, or a low max-snippet value tells a crawler not to show a quote from the page at all — which defeats the entire point if the goal is to be cited.
The same logic applies to anything gated behind a login, a cookie wall, or a script-only render with no fallback. A crawler that cannot reach the words cannot quote them, no matter how good the words are. Before spending more time writing the page, run it through your SEO audit template and confirm nothing between the crawler and the content is quietly saying no on your behalf.
Questions people also ask
Does blocking GPTBot remove me from ChatGPT search citations?
No. GPTBot is OpenAI's training crawler. The bot that builds the index behind ChatGPT's cited answers is OAI-SearchBot, and the one that fetches a page live during a chat is ChatGPT-User. Blocking GPTBot only keeps you out of future model training.
Does Google-Extended control whether I show up in AI Overviews?
No, and this is the most common mistake in this file. Google-Extended only controls training for the Gemini app and Vertex AI. Google Search, including AI Overviews and AI Mode, is driven entirely by Googlebot.
Do I need a separate robots.txt rule for every AI crawler?
You need one explicit User-agent block per crawler you want to name, even though a wildcard Allow already covers them by default. Explicit rules survive someone later adding a narrower Disallow rule that would otherwise catch them by accident.
Is robots.txt the only place an AI crawler can get blocked?
No. Hosting providers, CDNs and firewalls ship bot-management rules that block AI user agents by default, independent of your robots.txt file, and they fail silently with no error in your logs.
Should I block scrapers that ignore robots.txt anyway?
Robots.txt is a request, not a lock, so a scraper that ignores it needs a firewall rule, not a robots.txt line. Reserve robots.txt for the crawlers that respect it and handle abusive ones at the edge.
Sources
- Overview of OpenAI crawlers — OpenAI (checked 22 September 2026)
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic (checked 22 September 2026)
- Perplexity crawlers and bots — Perplexity (checked 22 September 2026)
- Overview of Google crawlers and fetchers — Google Search Central (checked 22 September 2026)
Find out which crawlers can actually reach your site
Book a free 30-minute consultation with Ibrahem and Jeremy. We look at your site while we talk, and you leave with a prioritised plan.
Who wrote this
Ibrahem HelalCo-founder
Leads strategy and delivery.
Works on: SEO, GEO
Reviewed before publishing by Jeremy Zava, Co-founder.
- Technical SEO: the complete checklist for 2026Technical SEO makes sure search engines and AI crawlers can reach, render and index your pages. This checklist covers every layer, in the order to fix it.
- Crawl Budget: What It Is and When It MattersCrawl budget is how many URLs a search engine is willing and able to crawl on your site in a given window. It matters for large sites, not small ones.
- Indexing Problems: Why Pages Stay OutPages stay unindexed for a handful of reasons: low quality, a blocked crawl path, a wrong canonical, or Google simply choosing not to bother yet.
- SEO Audit Template: What to Check and HowA working SEO audit template covering access, indexing, content, links and speed, with what to check in each layer and how to tell pass from fail.