Last updated: July 26, 2026 · By Vishal Swami, Founder & Lead AI Reviewer, AISagely
Cloudflare's new AI traffic options for customers let every site owner — free plan included — split AI bot traffic into three categories (Search, Agent, and Training) and decide separately whether to allow, block, or eventually charge each one. Cloudflare announced the change on July 1, 2026, and new defaults that block Training and Agent bots on ad-monetized pages take effect September 15, 2026.
Short answer: Cloudflare's new AI traffic options let any customer, including Free tier, control AI bots by three purposes — Search, Agent, and Training — instead of one blanket "block AI bots" toggle. Starting September 15, 2026, new domains default to blocking Training and Agent bots on pages with ads while still allowing Search. Existing sites can opt out any time before then in Security settings.
I run this site behind Cloudflare, so when this rolled out I went into my own dashboard the same week to see how it actually behaves, not just what the announcement post claims. Here's what changed, how to set it up, and where the new defaults can quietly break something you didn't mean to block.
What actually changed on July 1, 2026
Cloudflare's old "Block AI bots" managed rule treated AI crawlers as one lump category. In its July 1, 2026 blog post, Cloudflare split that into three distinct purposes a bot can be classified under:
- Search — a crawler building an index to answer queries later and send you referral traffic (think a classic search engine crawler).
- Agent — real-time automation acting on a person's behalf right now, like ChatGPT-User fetching a page or Gemini/Claude driving a browser.
- Training — a crawler pulling your content in to train or fine-tune a model, where your data gets permanently absorbed rather than just referenced.
The reasoning, per Cloudflare, is that a lot of real-world crawlers touch more than one of these — Googlebot, for instance, can be classified under both Search and Training — so the old single on/off switch couldn't express "let this bot index me for search, but don't let it train a model on my archive." The new controls can, and they're live now for every customer, including Free tier, in each zone's Security settings.
Cloudflare also introduced BotBase, a searchable directory of every known verified bot and its classifications, though that piece is Enterprise Bot Management only for now. And it extended the existing Content Signals robots.txt field with a new optional use directive — immediate, reference, or full — so a site can say a bot may index and link back to content, but not reproduce it in full.
What you'll need
You need an active Cloudflare zone (a domain on Cloudflare's network) — the Free plan works, since these controls are available on every plan. You'll want admin access to that zone's dashboard, since the toggles live under Security → Settings → AI Traffic Controls (Cloudflare also files it under the "AI Crawl Control" product tab). You don't need any code changes or a developer to flip the basic Allow/Block toggles; Cloudflare says the feature works with zero configuration required. If you want to go further and use the Pay Per Crawl beta, you'll additionally need a Stripe account connected on the payout side, which takes a few extra minutes. Nothing here requires downtime or a DNS change.
Step-by-step: setting Cloudflare's AI traffic options
1. Open Security settings for your zone
Log into the Cloudflare dashboard, pick the domain you want to configure, and go to Security. In my testing this is where the new AI traffic panel replaced the old single "Block AI bots" toggle — the three categories now show as separate switches instead of one row.
2. Set Search, Agent, and Training independently
Choose Allow or Block for each of the three categories. In my testing, leaving Search allowed while blocking Training kept normal search-engine indexing intact but stopped the crawlers Cloudflare classifies as training-only. If a bot is classified under more than one purpose, Cloudflare applies whichever rule is more restrictive — so blocking Training will also block a multi-purpose crawler like Googlebot if Google's crawler is tagged for both Search and Training on your traffic.
3. Decide whether to opt out of the September 15 defaults
If you're an existing customer and want to keep your current settings unchanged, mark that in Security settings before September 15, 2026 — Cloudflare has said it will keep notifying customers as that date approaches. If you do nothing and you're a new domain onboarding after that date, Training and Agent get blocked by default on any page that serves ads, while Search stays allowed.
4. Check your managed robots.txt for the new use signal
If you've already turned on Cloudflare's managed robots.txt, it already includes a Content Signals line like Content-Signal: search=yes,ai-train=no. Cloudflare is now layering in use=reference by default, meaning bots may index and excerpt your content but not reproduce it wholesale. You can check this under the AI Crawl Control tab's Directives section.
5. Try Pay Per Crawl if you want to charge for access (optional, beta)
Pay Per Crawl is still a private beta, per Cloudflare’s own docs, and it works by returning an HTTP 402 Payment Required response to crawlers you've chosen to charge instead of allow or block outright. The minimum price you can set is $0.01 USD per successful crawl (a 200 response), and you pick the price per bot or per path.
Robots.txt snippets you can copy
If you manage robots.txt yourself instead of using Cloudflare's managed version, here's the syntax straight from the new Content Signals extension. Add this line for your whole site to allow indexing but block both training and full reproduction:
“ User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / “
To be stricter and only allow a bot to interact without storing anything at all:
“ User-agent: * Content-Signal: search=yes,ai-train=no,use=immediate Allow: / “
Remember this is a signal, not an enforcement mechanism on its own — a well-behaved, Verified bot honors it, but robots.txt alone doesn't block traffic the way the dashboard's Allow/Block toggles do. Pair the two: robots.txt states your preference, the Security settings actually enforce it at the edge.
Common mistakes to avoid
The mistake I see most often is blocking Training and assuming search rankings are untouched — if the same crawler is tagged for both Search and Training (which Cloudflare says is common with mixed-use crawlers like Googlebot, Applebot, and Bingbot), blocking Training blocks that crawler entirely under the new rules, since the most restrictive category wins. Second, people set this once and forget the September 15 deadline — if you're happy with your current AI bot settings, you still need to actively opt out, because doing nothing means the new default applies to new domains and the "allowed by default" assumption for Verified bots no longer holds the way it used to. Third, don't confuse robots.txt Content Signals with actual blocking; it's a courtesy flag, and Cloudflare's real enforcement is the Security settings toggle. Fourth, don't enable Pay Per Crawl expecting meaningful revenue right away — it's a private beta with a one-cent minimum, and most AI crawlers you'd want to charge aren't required to pay it. Fifth, if you run ads on your site, check ad-serving and analytics bots aren't accidentally swept into a blanket "block everything AI" rule; they're generally classified outside the three core categories, but it's worth confirming in your traffic logs.
Search vs. Agent vs. Training: what each control actually does
| Category | Example behavior | Typical bot | Default after Sept 15, 2026 (new domains, ad pages) |
|---|---|---|---|
| Search | Builds an index to answer queries later, sends referral traffic | Classic search crawlers | Allowed |
| Agent | Real-time action on a person's behalf, right now | ChatGPT-User, browser-use agents (Gemini, Claude) | Blocked |
| Training | Collects content to train or fine-tune a model | Model-training crawlers | Blocked |
Tools that make this easier
Managing AI crawler access is really an SEO and content-visibility decision as much as a security one, and it overlaps with a lot of the AI tooling I already cover on this site. If you're weighing whether to allow Search bots so your content still surfaces in AI answers, my best AI tool for SEO optimization guide covers the tools that help you track that visibility. If you want to understand how AI tools evaluate and cite sources in the first place — relevant if you're deciding whether to allow Agent traffic from tools like ChatGPT or Gemini — see how to use ChatGPT search engine and how to use Gemini Live, both of which touch on how these tools fetch and cite live pages. Since ChatGPT-User and browser-driving agents fall under Cloudflare's new "Agent" category specifically, how to use ChatGPT agent mode is worth reading if you're deciding whether that traffic is worth allowing on your site. Claude's browser-use behavior falls in the same bucket — how to use Claude AI covers what that looks like from the user side. And if none of this is worth a paid tool to you yet, my free AI tools roundup and AI tool ratings page cover how to evaluate any AI product, including whether it's the kind of well-behaved, Verified crawler that's worth allowing through in the first place.
My take
The three-way split is the right call, and it's overdue — a single "block AI" switch never made sense once search visibility and model training became different questions with different answers for different site owners. What I'd watch closely is the September 15 default, specifically because it silently blocks any multi-purpose crawler the moment you block Training, including Googlebot if Google ever ships it as a combined crawler in your traffic. If you get real referral value from being indexed, don't set-and-forget this; check your Security settings before the deadline rather than trusting the default to match what you actually want.
Frequently Asked Questions
Is Cloudflare's new AI traffic options feature free to use?
Yes. The Search, Agent, and Training controls are available on every Cloudflare plan, including Free, at no extra cost. Pay Per Crawl, the optional monetization feature, is a separate private beta and isn't required to use the traffic controls.
When do the new default settings take effect?
September 15, 2026. That date only applies to new domains onboarding to Cloudflare after that point — existing customers keep their current AI bot settings unless they choose to change them, though Cloudflare will keep sending reminders as the date gets closer.
What happens to Googlebot if I block Training?
It depends on how Cloudflare's traffic classifies that specific crawler request. Cloudflare has said multi-purpose crawlers like Googlebot, Applebot, and BingBot are tracked under every purpose they perform, and the most restrictive rule you've set wins. If you block Training and Googlebot's crawl is tagged as both Search and Training, it gets blocked outright — not just for training purposes.
Do I have to pay to block AI crawlers?
No. Blocking is free and available on every plan. Pay Per Crawl is the opposite of blocking — it's for site owners who'd rather charge a crawler than block it outright, with a $0.01 USD minimum price per successful crawl, and it's still in private beta.
How is this different from the original Pay Per Crawl launch?
Pay Per Crawl (announced in 2025) was Cloudflare's monetization layer for AI crawlers and is still in private beta. The new AI traffic options announced July 1, 2026 are a separate, broader change: they replace the single "block AI bots" toggle with three independently configurable categories — Search, Agent, and Training — available to every customer for free, whether or not they ever touch Pay Per Crawl.