Cloudflare to Block AI Bots by Default, Test Pay-per-Crawl

Cloudflare is changing one of the web’s basic rules for dealing with bots: AI crawlers will be blocked by default on new domains that join its network. Site owners will be able to authorize them manually, keep them blocked or, in a later phase, require payment for each crawl.

The decision matters because of the company’s scale. Cloudflare protects and accelerates a significant share of global traffic and has become standard infrastructure for publishers, retailers and small websites. Its new setting could suddenly make it harder for AI developers to access large volumes of publicly available content.

From link exchanges to zero-click answers

For decades, the implicit deal between search engines and publishers was straightforward: search engines indexed a page and sent visitors in return. Those visitors could generate advertising revenue, subscriptions or exposure for whoever had published the content.

AI assistants break that cycle by synthesizing answers within their own interfaces. Users can get an explanation, a recipe or a summary without visiting the original source. Google has moved in the same direction with direct answers and AI Overviews.

Cloudflare says the imbalance is already measurable across its network. According to its data, getting referral traffic from OpenAI is 750 times harder for a creator than it was with Google’s earlier search engine; with Anthropic, the gap rises to 30,000 times. These calculations are based on the relationship between crawls and referral visits, not on a measurement of the revenue each publisher has lost, but they illustrate the problem: bots consume far more pages than they return in the form of audience.

Blocking is no longer the publisher’s job

Until now, preventing crawling required each website to configure technical rules, such as the robots.txt file, or identify and block specific addresses. The system had two shortcomings: not every publisher has the necessary technical resources, and not every bot respects those instructions.

Cloudflare is centralizing that decision. Customers will be able to choose from their dashboard whether AI bots can access their content, and the company will add mechanisms for those bots to identify themselves. The change applies by default to new customers; sites already hosted on Cloudflare will retain the ability to choose their settings.

The initiative has the backing of publishing groups including Associated Press, Condé Nast, The Atlantic, Fortune, Gannett, Hearst, TIME, Vox Media and Ziff Davis. For these groups, blocking is not merely a defensive tool: it strengthens their negotiating position with companies that need current, specialized and reliable data to train or improve their models.

Pay per Crawl aims to put a price on access

The second part of the announcement is Pay per Crawl, a proposal still in the experimental stage. The idea is for a publisher to set a price for each crawl request from an AI bot. If the developer agrees to pay it, access is granted; otherwise, the server can reject the request.

Cloudflare envisions the mechanism as a potential marketplace between content owners and AI companies. It is not intended to charge for each human reader or replace the direct licensing deals some major publishers already negotiate with labs such as OpenAI, but rather to create a standardized technical channel for automated access.

The most complicated question remains unresolved: how much each page is worth and who sets that value. Charging the same amount for a recycled article as for original investigative reporting does not appear sustainable. Cloudflare proposes that, over time, the price should reflect how much new information a piece contributes to AI systems, but that valuation is still far from becoming an operational standard.

Direct pressure on AI models

The default block will not prevent major labs from reaching commercial agreements with publishers or turning to licensed data, proprietary archives and user-generated content. But it does raise the cost of assuming that everything published openly is available to be collected without compensation.

The outcome will depend on two factors. The first is adoption: Cloudflare has reach, but it does not control the entire web. The second is whether developers accept a payment protocol instead of negotiating individual contracts or seeking alternative sources. If both sides sign on, AI crawling could stop being an invisible extraction process and become an explicit commercial relationship.