Cloudflare's AI crawler defaults flip today, and small sites may vanish from AI answers
As of September 15, 2026, Cloudflare blocks Agent and Training crawlers by default on ad-supported pages for new domains and every free-tier account that never changed its settings — reshaping which sites AI answer engines can see.
Cloudflare’s revised AI crawler defaults took effect today, blocking Agent and Training bots on ad-supported pages for new domains and every free-tier account that never touched its settings. The switch, announced July 1 by Cloudflare’s Jin-Hee Lee and Bryan Becker, quietly reshapes which sites AI answer engines can even see.
The mechanism is a three-part taxonomy: Search, Agent, Training. Traditional indexers like Googlebot, Applebot, and BingBot fall under Search. Anything multi-purpose gets judged by the most restrictive rule that applies, which means a site that blocks Training crawlers also blocks Google’s crawler wherever Google is doing more than pure Search. Cloudflare has framed the policy as overdue leverage against a lopsided arrangement in which, by its own telling, the world’s largest search engine has “2x more information” than rival AI companies because Search discoverability was historically bundled with training acceptance.
Prince’s other framing point is that bots have already surpassed human traffic online, sooner than the 2027 milestone the industry had penciled in. Cloudflare’s own data adds that more than 50% of AI crawler traffic is spent re-fetching pages that haven’t changed.
The commercial layer is Pay Per Use, the evolution of Pay Per Crawl, with Ceramic.ai and You.com as launch partners. Enterprise customers also get BotBase, a searchable directory of tracked bots, and new “content use” controls with immediate, reference, and full tiers.
“Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent,” Prince said.
What this collides with is the visibility economy those answer engines now define. NP Digital’s Fintech AI Visibility Index, analyzed across more than 68,000 AI-generated responses from ChatGPT, Gemini, Perplexity, Claude Sonnet 4, and Google AI Overviews, found Gemini names an average of 10 brands per response versus 6 for Perplexity. SoFi surfaced in nearly 50% of relevant prompts at an average first-mention position of 3.75, ahead of Bank of America at 6.07.
A site invisible to Agent traffic surfaces in none of them. The default has already been picked; the choice, block, allow, or monetize, was made on behalf of anyone who wasn’t watching.
Sources
- https://blog.cloudflare.com/content-independence-day-ai-options/
- https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/
- https://www.helpnetsecurity.com/2026/07/02/cloudflare-ai-crawler-controls/
- https://www.technology.org/2026/07/03/cloudflare-blocks-mixed-use-ai-crawlers/
- https://agilebrandguide.com/yesterdays-martech-ai-cx-news-september-15-2026/
— END —