The AI crawlers list: every bot to know
GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more: what each AI crawler does, what blocking it costs you, and how to check your rules.
The AI crawlers worth knowing are OpenAI's GPTBot, ChatGPT-User and OAI-SearchBot; Anthropic's ClaudeBot, Claude-User and Claude-SearchBot; Perplexity's PerplexityBot and Perplexity-User; Google's Googlebot plus the Google-Extended robots.txt token; and a longer tail from Apple, Meta, ByteDance, Amazon and others. Each does a different job, and blocking the wrong one is the most common way a brand disappears from AI answers.
This list is the reference version of AI search optimization: the crawler names change slightly as products evolve, but the underlying pattern — training bot, live-fetch bot, search-index bot, each needing its own decision — holds across every company in it.
On this page
The AI crawlers at a glance
| Crawler | Operator | Job | Costs you if blocked |
|---|---|---|---|
| GPTBot | OpenAI | Training data collection | Little — no visible effect on current citations |
| ChatGPT-User | OpenAI | Live fetch during a conversation | That page cannot be read when a user's question points at it |
| OAI-SearchBot | OpenAI | Search index behind citations | Removal from ChatGPT's citations generally |
| ClaudeBot | Anthropic | Training data collection | Little — no visible effect on current citations |
| Claude-User | Anthropic | Live fetch during a conversation | That page cannot be read in that conversation |
| Claude-SearchBot | Anthropic | Search index behind citations | Removal from Claude's citations generally |
| PerplexityBot | Perplexity | Retrieval index behind most citations | Removal from Perplexity's citations generally |
| Perplexity-User | Perplexity | Live fetch when a question points at a page | The freshest version of your content in that answer |
| Googlebot | Search crawl and index (Search, AI Overviews, AI Mode) | Removal from Google Search and its AI features entirely | |
| Google-Extended | robots.txt token for Gemini training/grounding | Nothing visible — no separate crawler exists |
OpenAI's three crawlers
GPTBot collects training data for future models; blocking it has no effect on what ChatGPT can cite today [1]. ChatGPT-User fetches a specific page live, in the moment a user's question calls for it — this is the one that lets a fresh, single-page lookup work inside a conversation. OAI-SearchBot builds the search index behind ChatGPT's broader citations, the one most GEO work is actually trying to win.
Treat these as three separate decisions rather than one "allow OpenAI" toggle. A site that allows GPTBot but blocks the other two is contributing to training data with nothing to show for it in citations; a site that blocks all three has opted out of ChatGPT visibility entirely, whether or not that was the intent.
OpenAI's own documentation names all three explicitly and updates the page when a new one launches, which makes it the source to check rather than a third-party summary — including this one — whenever a rule needs to be written with confidence [1].
Anthropic's three crawlers
The pattern repeats with different names. ClaudeBot handles training; Claude-User does a live fetch tied to one conversation; Claude-SearchBot builds the index behind Claude's own web search feature, and Anthropic's support documentation is explicit that these do different jobs with different consequences when blocked [4].
Claude is the most conservative of the major assistants about what it will cite at all, so losing Claude-SearchBot access costs a scarcer, more durable kind of visibility than losing the equivalent crawler elsewhere — how Claude chooses its sources covers that scarcity in more depth.
Perplexity's two crawlers
Simpler by count but no less important: PerplexityBot builds the retrieval index behind most of Perplexity's citations, and Perplexity-User does the live fetch [3]. Perplexity cites more sources per answer than any other major assistant, so a page open to both of these has a wider door to walk through than the same page would have with ChatGPT or Claude.
Google's crawlers, and the Google-Extended exception
Googlebot is the one crawler on this list that, if blocked, costs you nearly everything: normal Search, AI Overviews and AI Mode all depend on the same crawl and index [2]. Google-Extended is different in kind rather than degree — it has no separate user agent, is evaluated only after Googlebot has already crawled a page, and controls Gemini training and grounding alone, with no effect on Search or AI Overviews inclusion [2].
Confusing the two is the single most common mistake on this whole list: teams disallow Google-Extended believing it is Google's version of "block AI crawlers" and change nothing about their actual AI visibility, because Googlebot access was never the thing they touched.
The other crawlers worth a deliberate decision
Apple's Applebot-Extended governs training data for Apple's own AI features, on the same non-fetching, robots.txt-token model as Google-Extended. ByteDance's Bytespider and Common Crawl's CCBot both feed training datasets used well beyond their original operators, including by some smaller AI products that license or scrape from Common Crawl's public archive rather than crawling directly. Amazonbot supports Alexa and Amazon's own AI features. Meta's crawler collects training data for Meta AI, and DuckDuckGo's DuckAssistBot supports its AI-powered search answers.
None of these carry the same citation stakes as the OpenAI, Anthropic, Perplexity or Google rows above for most businesses today, but "carries less stakes" is not the same as "safe to ignore." A blanket bot-blocking rule copied from a security template can catch these along with the ones that matter, and nobody notices a gap in a crawler that was never being tracked in the first place.
A reasonable default for this second tier: allow the ones tied to a product your buyers might actually use to ask about you — Amazonbot if you sell where Alexa shopping questions are plausible, DuckAssistBot if your audience skews toward privacy-focused search — and treat the rest as training-only unless you have a specific reason to think otherwise.
How do you actually allow or block one of these?
Start with robots.txt
Write an explicit `User-agent` block naming each crawler by its exact string, rather than relying on a wildcard rule someone else wrote. Check each company's own published bot page for the current name before you write the rule.
Check your edge and WAF separately
A network-level bot-protection product, whether it is Cloudflare's or another vendor's, applies its own decision independent of robots.txt and can silently override a correctly written file [5].
Decide per crawler, not per company
Blocking a company's training bot while allowing its retrieval and search-index bots is a defensible, common choice. Treating all of a company's crawlers as one switch usually costs more than it saves.
How do you verify your rules actually work?
Fetch your own pages under each crawler's exact user agent string and read the response, rather than trusting what the robots.txt file or the dashboard toggle says should happen. This catches the gap between intent and reality: a rule that reads correctly on the page but gets overridden by a firewall further along the request path.
Repeat the check after any change to hosting, security plugins, or bot-management settings, and again on a quarterly cadence even with nothing to prompt it, because this list itself will be out of date by the time a new AI product launches with its own crawler. Our AI visibility audit runs this exact fetch test across every engine in one pass, along with a real prompt set to confirm the access actually translates into citations, which the ranking factors that decide a citation and how each engine compares on search behaviour cover from the content side rather than the crawler side. Getting cited by Gemini specifically is the natural next read once your crawler access across the board is confirmed.
Questions people also ask
Which AI crawlers should every business allow by default?
The retrieval and search-index crawlers from any engine you want to be cited by: ChatGPT-User and OAI-SearchBot, Claude-User and Claude-SearchBot, and Perplexity-User and PerplexityBot. Training-only crawlers like GPTBot and ClaudeBot are a separate, lower-stakes decision.
Do these crawlers respect robots.txt?
The major ones from OpenAI, Anthropic, Perplexity and Google state that they do. Some smaller or less transparent AI scrapers do not, which is why a network-level control like Cloudflare's AI Crawl Control matters alongside robots.txt, not instead of it.
Is Google-Extended a crawler I need to allow or block?
No — it has no separate crawler or user agent. It is a robots.txt token read after Googlebot's own crawl, and it only controls Gemini training and grounding, not Search or AI Overviews inclusion.
How often does this list change?
Often enough to check quarterly. New AI products add new crawlers, and a rule copied from an old checklist can miss one entirely — verify against each company's own published bot page rather than a third-party list, including this one, before relying on it long-term.
Sources
- Overview of OpenAI crawlers — OpenAI (checked 23 September 2026)
- Overview of Google crawlers and fetchers — Google Search Central (checked 23 September 2026)
- Perplexity crawlers — Perplexity (checked 23 September 2026)
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic (checked 23 September 2026)
- AI Crawl Control — Cloudflare (checked 23 September 2026)
Find out which AI crawlers can actually reach your site today
Book a free 30-minute consultation with Ibrahem and Jeremy. We look at your site while we talk, and you leave with a prioritised plan.
Who wrote this
Jeremy ZavaCo-founder
Leads growth and client partnerships.
Works on: CRO, SEO
Reviewed before publishing by Ibrahem Helal, Co-founder.
- AI search optimization: the 2026 guideHow to get cited by ChatGPT, Gemini, Perplexity, Claude, Copilot and Google's AI features: crawler access, citable pages, third-party proof and measurement.
- ChatGPT Search vs Google: What's DifferentGoogle still handles the overwhelming majority of searches worldwide, but ChatGPT wins a specific kind of question — and the two barely cite the same pages.
- AI Search Ranking Factors That MatterAI engines do not use one ranking algorithm, but the pages they all reach for share nine traits: primary, specific, fluent and easy to lift.
- How to Get Cited by Google GeminiGemini leans on forums and roundups and mentions brands far more than it links them — here's how to close that gap for yours.