Cloudflare's Default Flip Takes Effect Today, Reshaping AI…

As of September 15, Cloudflare blocks Training and Agent crawlers by default on ad-bearing pages for all new domains and existing free-tier customers who never changed their settings.

Cloudflare’s revised bot defaults take effect today, and any free-tier domain that ignored the July 1 announcement is now blocking Training and Agent crawlers on ad-bearing pages by default. Search crawlers stay allowed. The two-and-a-half-month opt-out window between the July 1 policy release and September 15 is closed, and inaction is a decision.

The taxonomy is the mechanism. In a July 1 blog post, Cloudflare split crawler behavior into eleven categories, with three doing most of the work: Search indexes content to answer later queries, Agent acts in real time on a person’s behalf, and Training absorbs content permanently into a model. Transact, Data Collection, SEO, Ads Verification, Social/Link Preview, Feed Fetching, Security Testing, and Monitoring & Operations round out the list. Mixed-purpose crawlers that refuse to let site owners separate the three headline categories are blocked outright on pages with ads.

That last clause has teeth. Per Cloudflare’s developer documentation, any configuration that blocks Training also blocks crawlers combining Search and Training, which, as Help Net Security noted, catches Googlebot, Applebot, and BingBot on sites that toggle the legacy “Block AI bots” option. Google’s response has been to point at its Google-Extended token, which lets sites opt out of Gemini Apps and Vertex API training without touching Search inclusion. Cloudflare’s counter, relayed through TechCrunch, is that the “world’s largest search engine” already has roughly 2x more information than other AI companies because it ties discoverability to AI use.

CEO Matthew Prince framed the goal as pressure: to “encourage mixed use crawlers to separate out search from agent use and training.” He also offered the structural read: “now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.” Cloudflare says automated traffic crossed the 50% threshold a year ahead of its own forecast, and that over half of AI crawl volume re-fetches unchanged pages.

Around the default sits a market. Cloudflare’s Pay Per Use program, an evolution of Pay Per Crawl, launched alongside partners including Ceramic.ai, which pays publishers when their content surfaces in AI results, and You.com, which lets an agent pay on demand for a specific page. Cloudflare cites more than 50 major licensing deals signed in the past year.

The 2015 net-neutrality fights were about who could reach the user. This one is about who the user’s agent can reach.

Cloudflare's Default Flip Takes Effect Today, Reshaping AI Discovery for Free-Tier Sites

Sources