Cloudflare sets September deadline to block AI agents by default on ad-supported pages
On its second 'Content Independence Day,' the company split crawlers into Search, Agent, and Training — and opened a Monetization Gateway that charges AI bots in stablecoins over the x402 protocol.
Cloudflare will begin blocking AI Agent and Training crawlers by default on September 15, 2026 for any page that displays ads, the company announced Wednesday on the second anniversary of what it calls Content Independence Day. The rule applies to new domains, new sites launched by existing customers, and every Free-tier account that hasn’t manually overridden its settings.
The mechanism is a new three-way taxonomy. Cloudflare is splitting crawlers into Search, Agent, and Training buckets, with only Search allowed to pass through by default. Agents are defined by the company as automation “acting, usually in real time, on a person’s behalf.” Training bots are what they sound like. The taxonomy is available to all customers, including Free.
The interesting part is what happens to crawlers that refuse to sort themselves. Mixed-use bots such as Googlebot, Applebot, and BingBot get blocked entirely if a site opts out of Training, because Cloudflare can’t parse which fetches feed search indexes and which feed model weights. That’s a deliberate squeeze: separate your crawlers into labeled bots, or lose search visibility along with training access.
The pressure is legible in Cloudflare’s own bot report. AI training accounted for 52% of crawler requests in June 2026, up from 22% in spring 2025, with mixed-use crawlers responsible for more than 36% of activity. “now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said co-founder and chief executive Matthew Prince.
Alongside the blocking regime, Cloudflare opened a waitlist for its Monetization Gateway, which lets publishers charge bots for content, APIs, and MCP tools in stablecoins via the x402 protocol. The old Pay Per Crawl experiment is being folded into a broader Pay Per Use model. Ceramic.ai and You.com are the launch partners. A separate Attribution Business Insights dashboard will show publishers what AI bots are consuming and what human referrals they generate in return, and Verified Bots that ignore declared preferences or reproduce content wholesale can lose verified status.
Cloudflare is doing what the 1998 robots.txt handshake never had teeth to do: turning a polite request into a toll booth, with a settlement layer attached.
Sources
- https://www.cloudflare.com/press/press-releases/2026/cloudflare-allows-the-agentic-internet-to-flourish-with-a-simple-philosophy-your-content-your-rules/
- https://blog.cloudflare.com/content-independence-day-ai-options/
- https://blog.cloudflare.com/agentic-internet-bot-report/
- https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/
- https://www.helpnetsecurity.com/2026/07/02/cloudflare-ai-crawler-controls/
— END —