Cloudflare AI bot blocking: what it does
Cloudflare's AI Crawl Control toggle blocks AI crawlers site-wide in one click, including some that let you get cited. What it blocks, and how to fix it.
Cloudflare's AI bot toggle, part of AI Crawl Control, blocks the AI crawlers Cloudflare has identified — training bots, retrieval bots and AI search bots — at the network edge before they reach your server. Turning it on with no further setup blocks all of them, including the ones that let ChatGPT, Perplexity and Claude cite your pages.
That last part is the trap. If your goal is AI search optimization rather than keeping every AI system off your site, the default toggle is close to the opposite of what you want, and the fix is to go bot by bot rather than to use the single switch.
On this page
What does Cloudflare's AI Crawl Control actually do?
It is a network-level filter that sits in front of your site and inspects every request's user agent and source. When a request matches a crawler on Cloudflare's maintained AI bot list, AI Crawl Control can block it, allow it, or log it, before your origin server ever sees the request [1].
That is different from robots.txt, which is a text file a crawler can choose to read and honour. Cloudflare's control does not depend on the crawler's goodwill: a bot that ignores robots.txt entirely still gets the edge decision applied to it, because the block happens on Cloudflare's network, not on trust [2].
The single "Block AI Bots" switch in the dashboard applies one decision — block — to every crawler on that list at once. That is the convenient part and the dangerous part in the same feature.
Cloudflare built the feature in response to a real problem: AI companies were scraping sites at a scale that cost site owners real bandwidth and server load, often with no way to opt out short of blocking everything from that vendor. The blanket switch solves that problem completely. It was never built to distinguish between a crawler that only trains a model and one that is, right now, the reason a specific answer names you, so it treats them the same unless you go in and separate them yourself.
How does this interact with robots.txt if you already have AI crawler rules there?
It layers on top, and the two rarely agree by accident. A robots.txt file you wrote — or one a previous developer wrote — might already allow GPTBot and disallow nothing else, while Cloudflare's edge toggle blocks it regardless of what the text file says, because the edge decision is enforced before the request reaches the server that would have served robots.txt in the first place.
That ordering matters. A well-behaved crawler checks robots.txt and, finding itself allowed, tries to fetch the page — and gets a block response from Cloudflare anyway. From the crawler's side this looks identical to a server error or a dead site, and there is nothing in your robots.txt that would tell you why. Treat AI Crawl Control as the rule that actually governs the outcome, and robots.txt as a second, independent statement of the same intent, not a substitute for checking the edge settings.
Which crawlers does the default toggle cover?
Cloudflare groups recognised AI crawlers by what they do: bots that fetch content to train a model, bots that retrieve a page live to answer one question, and bots that build a search index behind an AI product's citations [1]. The one-click toggle does not distinguish between these groups — it blocks all three.
| Crawler type | Example | What blocking it costs you |
|---|---|---|
| Training crawler | GPTBot, ClaudeBot | Keeps your content out of future model training runs |
| Retrieval / live-fetch bot | ChatGPT-User, Claude-User, Perplexity-User | Removes you from answers to questions asked about your specific page right now |
| AI search-index bot | OAI-SearchBot, PerplexityBot, Claude-SearchBot | Removes you from that engine's citations entirely, not just one feature |
Blocking training bots costs you little if you were never trying to be training data. Blocking the second and third rows costs you the exact outcome a GEO programme exists to win, which is why the toggle needs a second look before you flip it.
How do you turn it on without losing citations you want?
Open the per-bot list before the summary toggle
Cloudflare's AI Crawl Control screen lists every recognised crawler individually with its own allow or block state. Start there, not on the single switch at the top.
Block training bots by default
If you have no reason to let your content train a model you get nothing back from, blocking GPTBot, ClaudeBot and their peers in this category costs you nothing visible.
Allow retrieval and search-index bots for engines you want to be cited by
Explicitly allow ChatGPT-User, OAI-SearchBot, Perplexity-User, PerplexityBot, Claude-User and Claude-SearchBot unless you have a specific reason to exclude one of them.
Re-check after any Cloudflare plan or ruleset change
New crawlers get added to Cloudflare's list over time, and a ruleset copied from another site can carry a blanket block you did not choose. Review the list again after any change to bot management or WAF rules.
Should you block every AI crawler, or allow some on purpose?
Not every crawler deserves the same answer, and "block everything AI-related" is a policy from before AI answers were sending anyone traffic. Answer per bot, based on what you actually get from being reachable by it: citations and, for engines like Perplexity that show numbered source links, real click-through traffic.
A useful default: allow anything that can cite you in an answer, block anything that only trains a model with no visible return, and decide about the rest case by case. How each engine picks its sources differs enough that a rule tuned for one does not automatically fit another.
Sites with a strong reason to opt out entirely — a subscription product whose whole value is the content itself — are the exception, and even then, blocking the retrieval bots costs you visibility that a training-bot block alone would not.
Local and multi-location businesses have a specific reason to keep the retrieval and search-index rows open: the engines answering "who should I call" questions can only name a business whose pages they can actually reach. Blocking the wrong bot here does not just cost a citation on a generic query; it removes you from exactly the buying-decision questions a GEO programme is built to win, at the point in the funnel where a competitor's open crawler wins the mention instead.
What does getting this wrong actually cost you?
Nothing you can see immediately, which is what makes the mistake last. A blocked retrieval bot produces no error message, no drop visible in standard analytics, and no alert — the crawler simply stops fetching your pages, and any answer engine that relied on it stops citing you from that point forward.
The realistic way this surfaces is a founder noticing, months later, that a competitor gets named in ChatGPT or Perplexity for a question the founder's own company should win on the facts. Tracing that back to a Cloudflare setting flipped during an unrelated security review is possible, but it takes someone knowing to look there, which most teams do not.
How do you verify the block, or the allow, actually worked?
Checking the dashboard toggle's state is not verification; it tells you what you intended, not what happened. Fetch a real page under each crawler's exact user agent string and confirm the response: a 200 with full HTML for anything you meant to allow, a block response for anything you meant to stop.
Do this after every change, not just the first time. A WAF rule, a firewall managed ruleset, or a hosting provider's own bot-protection layer can silently override what AI Crawl Control decided, and none of those produce an error a person notices — traffic for that crawler just stops.
What if Cloudflare is not your only line of defence?
Cloudflare's edge decision is upstream of your server, but it is not the only layer a request passes through. A hosting platform's own bot mitigation, a WAF product from a different vendor, or a CMS plugin that blocks by user agent can each undo what you configured at Cloudflare, and they rarely log it anywhere you would think to check.
Treat crawler access as one setting that lives in several places rather than one you set once. When something changes — a new host, a new security plugin, a new WAF ruleset — re-run the fetch test from the verification step above before assuming access is still what you configured.
Questions people also ask
Does Cloudflare's AI bot toggle block Googlebot too?
No. It targets crawlers Cloudflare has identified as AI-related — training, retrieval and search bots from OpenAI, Anthropic, Perplexity and others. Googlebot and Bingbot are classed separately and are not touched by this toggle.
Will turning it on remove me from ChatGPT or Perplexity answers?
If you block a retrieval or search crawler like OAI-SearchBot or PerplexityBot, yes — that engine can no longer read your pages to answer a question. Turning on the blanket toggle without reading the list first is the single most common way a brand disappears from AI answers without anyone noticing for months.
Is a Cloudflare toggle the same as a robots.txt rule?
No, and that is the point of having both. Cloudflare's control sits at the network edge and returns a block before the request reaches your server, whether or not the bot respects robots.txt. A robots.txt rule is a request a well-behaved crawler chooses to honour.
How do I know which crawlers my current settings block?
Cloudflare's dashboard lists every crawler it recognises with a per-bot allow or block state, and it logs blocked requests. Read that list bot by bot rather than trusting the summary toggle's on/off state alone.
Sources
- Declaring your AIndependence: block AI bots, scrapers and crawlers with a single click — Cloudflare (checked 23 September 2026)
- AI Crawl Control — Cloudflare (checked 23 September 2026)
- Overview of OpenAI crawlers — OpenAI (checked 23 September 2026)
Find out which AI crawlers can actually reach your pages
Book a free 30-minute consultation with Ibrahem and Jeremy. We look at your site while we talk, and you leave with a prioritised plan.
Who wrote this
Ibrahem HelalCo-founder
Leads strategy and delivery.
Works on: SEO, GEO
Reviewed before publishing by Jeremy Zava, Co-founder.
- AI search optimization: the 2026 guideHow to get cited by ChatGPT, Gemini, Perplexity, Claude, Copilot and Google's AI features: crawler access, citable pages, third-party proof and measurement.
- How Google AI Overviews choose sourcesAI Overviews break your query into sub-questions, then cite whichever pages answer each part cleanly — length and top-10 rank barely matter.
- Get ChatGPT to Recommend Your BusinessChatGPT builds local recommendations from what it reads about you online, not a paid listing, so accurate, consistent hours and address matter most.
- The AI crawlers list: every bot to knowGPTBot, ClaudeBot, PerplexityBot, Google-Extended and more: what each AI crawler does, what blocking it costs you, and how to check your rules.