Cloudflare just split AI access into three switches. Flip the wrong one, and you vanish from search.
Crawl access is now three decisions: Cloudflare splits AI traffic into Search, Agent, and Training, each managed individually.
Blocking Training can block Googlebot: under the new “strictest-rule” logic, multi-purpose bots default to the most restrictive setting.
The exposure is narrow but real: it centers on legacy “Block AI Bots” adopters and unconfigured free-tier sites.
Agent access is a conversational-visibility lever: blocking it means opting out of ChatGPT’s and Perplexity’s live answers, independent of Google.
Executive Summary: Cloudflare reworked how it classifies and gates AI crawlers, fracturing the permission decision between Search, Agent, and Training (bots). Starting September 15, blocking Training can silently block Googlebot, turning a defensive content-protection setting into a search-visibility liability. Crawler access is no longer an IT footnote, but a commercial control that determines whether a brand is even eligible for AI citation.
On July 1, 2026, Cloudflare retired its one-click “block AI bots” toggle, replacing it with behavior-based controls.
Then, it set new defaults that can quietly block crawlers.
If your brand operates on Cloudflare, this is the moment to revise your security settings – and to do so deliberately.
Cloudflare now sorts AI crawlers into three behaviors: Search, Agent, and Training.
The new controls let websites independently manage automated traffic for each behavior, and they are live for all customers, including the free tier.
Starting September 15, 2026, Cloudflare is introducing a new set of default settings:
The reason is structural: Google uses a single, mixed-use bot for both search indexing and AI training, so it automatically falls under the “strictest-rule” umbrella – and since they refuse to split its crawler, blocking Training at the edge means blocking Googlebot outright.
Even if you disallow Google-Extended in your robots.txt, it makes no practical difference. A Cloudflare block operates at the network level, making it far harder to bypass than a simple robots.txt line, which is an advisory instruction that Google can ignore.
Put bluntly, when it comes to Googlebot, there’s currently no clean way to allow search while opting out of training.
Because Googlebot only drops when a Training block is in effect, the real exposure narrows down to two groups: sites that flipped the legacy “Block AI Bots” toggle back in 2025 and free-tier accounts that never configured their crawler settings.
The first group is most at risk, purely because the legacy block is now folded under the “strictest-rule” regime. The second group is partially protected, since new defaults only block crawlers on pages that display ads, meaning that an ad-free site with no existing block gets to keep Google on the guest list.
So, for most brands, the honest takeaway is reassuring: you’re probably fine – but check anyway.
Come September 15, “politely declining” the use of your content for AI training may actually escalate into firewalling Googlebot – and that means your site won’t be crawled as effectively, which could eventually impact your visibility in search results. Fortunately, the fix for this one is a simple five-minute Security settings review: open Cloudflare dashbox and confirm, explicitly, that Search is allowed even where Training is blocked.
From the optimization perspective, Cloudflare effectively branched crawl access into three separate decisions, each of which maps cleanly onto visibility outcomes:
At its core, this is an eligibility problem before it’s a tactics problem. Put simply, no amount of structure or schema earns a citation if the network layer bounces the crawler away at the door. Therefore, verifying edge-level crawler reachability must become a part of your technical audit checklist – right alongside the crawl-access and indexability checks.
What’s more, there’s a flip side to that coin: Cloudflare is now laying the groundwork to charge AI crawlers rather than simply bar them – and if it takes, crawl access may become a veritable revenue decision.
| Business type | SearchIndexes for AI answers | AgentLive fetch per user | TrainingFeeds model training |
|---|---|---|---|
| B2B SaaS / services | AllowCitation eligibility | AllowChatGPT / Perplexity reach | Your callLow visibility cost |
| Ad-supported publisher | AllowKeep discoverability | Your callDefault blocks on ad pages | Block OKYour IP leverage |
| E-commerce / retail | AllowProduct visibility | AllowAgentic shopping | Your callLow stakes |
| Local / home services | AllowLocal AI Overviews | AllowAssistant reach | Your callMinimal concern |
The Google trap: blocking Training also blocks Googlebot. Google uses one bot for search and AI training, so there is no way to keep Google search visibility while blocking Google training at the network level.
Before September 15, 2026: confirm Search is allowed, and check for a legacy “Block AI Bots” toggle — it now maps to a Training block that drops Googlebot.
Jordan Parkes, the CEO and founder of ZeroClick Labs, has been building and scaling digital marketing strategies since 2012, leveraging performance-driven SEO and data-based digital marketing solutions to guide the growth of hundreds of companies across the U.S. and Europe.
Your misconfigured settings are turning those assistants away…
…and your buyers with them.
ZeroClick Labs offers a solution to that problem – and more.
We work the full GEO/SEO stack – from crawl-layer access to citation-first content clusters, across every frontier platform, including ChatGPT, AIOs, and Perplexity.
“Our agency had no idea how to approach AI visibility. ZeroClick only does this one thing so they actually know what works. Worth every penny just to not waste time figuring it out ourselves.” – Jay