Cloudflare has replaced its single "Block AI Bots" switch with three separate ones, and the three are not interchangeable. Search bots index you and send people back. Agent bots fetch a page in real time because a user asked something. Training bots take the text to build a model. From 15 September 2026, new domains on Cloudflare will have Training and Agent blocked by default on ad-monetised pages, with Search left on. Everyone else keeps whatever they had, which for most small sites means everything is allowed. The controls are free on every plan. Setting them takes about twenty minutes, and exactly one of the three decisions carries a real cost, which is where to start.
The Googlebot problem is the whole decision
Cloudflare's announcement states it plainly: multi-purpose crawlers that combine Search with Training, Googlebot among them, are blocked when Training is restricted. That is not a Cloudflare quirk, it is the shape of the market. The company that sends you search traffic is the same company training on the same fetch, and the crawler does not offer to separate the two.
So the honest framing is not "should I block AI training." It is: what fraction of your traffic arrives through the crawler you would have to block to stop the training. For a documentation site that gets found through search, that number is most of it. For a newsletter with a mailing list and a Discord, it may be almost none. Answer that first and the other two switches take a minute each.
What to set, by site type
These are our recommendations, not Cloudflare's, and the reasoning is stated so you can disagree with it on the parts that do not match your traffic.
| Site type | Search | Agent | Training | Why |
|---|---|---|---|---|
| SaaS marketing site or docs | Allow | Allow | Allow | Being cited by an assistant is a lead. Blocking Training costs you Googlebot, and organic search is how buyers find docs. |
| Publisher earning from ads or subscriptions | Allow | Consider blocking | Block | Agent fetches substitute for the visit that pays you. This is the case Cloudflare's September default is built around. |
| E-commerce catalogue | Allow | Allow | Allow | Agents increasingly shop. A blocked product page is a product that cannot be recommended, and your prices are public anyway. |
| Paid course, member area, proprietary research | Allow the marketing pages | Block | Block | The content behind the wall is the product. Accept the Googlebot loss on the gated paths only, using a path rule rather than a site-wide block. |
The fourth row is the one that needs the extra step. Cloudflare's per-crawler toggles apply to the zone; path-level exceptions are made with WAF rules, which is documented as the way to carve out sections rather than deciding for the whole domain.
The twenty-minute setup
- Look before you touch anything. In the Cloudflare dashboard, select your account and domain, open AI Crawl Control, then the Crawlers tab. The table lists each crawler with its operator, its category (AI crawler, AI assistant, archiver), request counts with trend charts, and recorded robots.txt violations. Spend five minutes here before changing a setting. Most people discover their traffic is dominated by two or three crawlers they had never heard of, and that the ones they were worried about are barely present.
- Decide per crawler, in the Actions column. Allow or Block is set per row. After allowing a crawler you can still choose Enforce robots.txt, which is the middle setting worth knowing about: access permitted, but your stated rules actually applied rather than treated as a suggestion.
- Set the block response. In the Settings tab, under Block response, choose the HTTP status and message. Paid plans can return 402 Payment Required instead of 403 Forbidden, which is the door to the Pay Per Crawl marketplace, currently a closed beta. Returning 402 today mostly signals intent, so do not build revenue plans on it.
- Do not confuse the signal with the block. Cloudflare says it outright: "content signals express preferences; they are not technical countermeasures against scraping," and some companies will ignore them. The robots.txt line is the stated reservation of rights. The Crawlers tab is the enforcement. You want both, and you should expect only the second one to work on a crawler that has decided not to care.
Publish your preferences in robots.txt. Cloudflare's Content Signals Policy adds a machine-readable line separating access from use. The syntax is a single directive:
Content-Signal: search=yes, ai-train=noThe three signals are defined precisely. search covers "building a search index and providing search results." ai-input covers "inputting content into one or more AI models," which is retrieval-augmented generation and AI answer grounding. ai-train covers "training or fine-tuning AI models." Cloudflare has already applied search=yes, ai-train=no to more than 3.8 million domains using its managed robots.txt, deliberately leaving ai-input unset rather than guessing. The policy is CC0, so you can paste it whether or not you use Cloudflare.
One caveat that applies to the Free plan specifically
On the Free plan, crawler detection relies on user agent strings only. A crawler that identifies itself honestly gets caught; one that presents as a browser does not. That is a meaningful limitation if your motivation is stopping a specific bad actor rather than expressing a policy to the ones that behave. If the goal is the latter, which it is for most sites, the free tier does the job.
What to check a week later
Come back to the Crawlers tab and read two columns. First, request volume: if a crawler you blocked is still generating requests, they are being refused, which is the system working, not failing. Second, the robots.txt violations count. That column is the most useful thing on the page, because it names which operators fetch paths you told them not to. A crawler with a rising violation count is telling you that its stated compliance is decorative, and it belongs on Block rather than Allow-with-enforcement.
Then check the other side of the ledger. If you blocked Training and therefore Googlebot, your impressions in Search Console will start falling within days, and that is the cost you agreed to pay. Watch it against the measurement routine we laid out for separating AI-driven click loss from ordinary ranking movement, because otherwise you will misread your own experiment. If the drop is bigger than you priced, reverse the switch. It is a toggle, not a migration.
The broader point is that this is a rare case of a platform handing over a control rather than exercising it. The September default only touches new domains, so nothing changes for an existing site until someone chooses. Choosing badly costs search traffic; not choosing at all means the answer stays "everything, to everyone, for free," which is a decision too. Write down which of the three you are trading away and why, add it to your dependency notes, and revisit it when the crawler mix in that table looks different from today.
Discussion
Sign in with Google or just a name. No email link, no password to remember.