Cloudflare to Block AI Bots by Default, Test Pay-per-Crawl

On July 1, 2025, Cloudflare announced default blocking of AI crawlers for new domains and introduced control options. The original source supports the documentary core of the event; a network setting does not itself change the law or prevent every access method.

The decision matters because of the company’s scale. Cloudflare protects and accelerates a significant share of global traffic and has become standard infrastructure for publishers, retailers and small websites. Its new setting could suddenly make it harder for AI developers to access large volumes of publicly available content.

From link exchanges to zero-click answers

For decades, the implicit deal between search engines and publishers was straightforward: search engines indexed a page and sent visitors in return. Those visitors could generate advertising revenue, subscriptions or exposure for whoever had published the content.

AI assistants break that cycle by synthesizing answers within their own interfaces. Users can get an explanation, a recipe or a summary without visiting the original source. Google has moved in the same direction with direct answers and AI Overviews.

Cloudflare says the imbalance is already measurable across its network. According to its data, getting referral traffic from OpenAI is 750 times harder for a creator than it was with Google’s earlier search engine; with Anthropic, the gap rises to 30,000 times. These calculations are based on the relationship between crawls and referral visits, not on a measurement of the revenue each publisher has lost, but they illustrate the problem: bots consume far more pages than they return in the form of audience. Document supporting the figure.

Blocking is no longer the publisher’s job

Until now, preventing crawling required each website to configure technical rules, such as the robots.txt file, or identify and block specific addresses. The system had two shortcomings: not every publisher has the necessary technical resources, and not every bot respects those instructions.

Cloudflare is centralizing that decision. Customers will be able to choose from their dashboard whether AI bots can access their content, and the company will add mechanisms for those bots to identify themselves. The change applies by default to new customers; sites already hosted on Cloudflare will retain the ability to choose their settings.

The initiative has the backing of publishing groups including Associated Press, Condé Nast, The Atlantic, Fortune, Gannett, Hearst, TIME, Vox Media and Ziff Davis. For these groups, blocking is not merely a defensive tool: it strengthens their negotiating position with companies that need current, specialized and reliable data to train or improve their models.

Pay per Crawl aims to put a price on access

The second part of the announcement is Pay per Crawl, a proposal still in the experimental stage. The idea is for a publisher to set a price for each crawl request from an AI bot. If the developer agrees to pay it, access is granted; otherwise, the server can reject the request.

Cloudflare envisions the mechanism as a potential marketplace between content owners and AI companies. It is not intended to charge for each human reader or replace the direct licensing deals some major publishers already negotiate with labs such as OpenAI, but rather to create a standardized technical channel for automated access.

The most complicated question remains unresolved: how much each page is worth and who sets that value. Charging the same amount for a recycled article as for original investigative reporting does not appear sustainable. Cloudflare proposes that, over time, the price should reflect how much new information a piece contributes to AI systems, but that valuation is still far from becoming an operational standard.

Direct pressure on AI models

The default block will not prevent major labs from reaching commercial agreements with publishers or turning to licensed data, proprietary archives and user-generated content. But it does raise the cost of assuming that everything published openly is available to be collected without compensation.

The outcome will depend on two factors. The first is adoption: Cloudflare has reach, but it does not control the entire web. The second is whether developers accept a payment protocol instead of negotiating individual contracts or seeking alternative sources. If both sides sign on, AI crawling could stop being an invisible extraction process and become an explicit commercial relationship.

Turning the headline into a check

Policy reading begins with authority. Separate statute, regulation, guidance, plan, contract and a platform's private setting. Then identify who is bound, from which date and before which body. a network setting does not itself change the law or prevent every access method. Without that classification, a policy priority can be misreported as an enforceable right or prohibition.

The text also needs an implementation chain: published rule, technical specification, budget, owner and evidence of compliance. The existence of a duty does not show that every tool can perform it or that enforcement detects every breach. To assess how to check scope, identified bot, technical rule and actual effect on a site, look for the points where the chain can break.

Exceptions and transitions are part of the rule. Older products, smaller actors, research, security or market dates may receive different treatment. A practical check preserves the applicable article or section and explains why the case belongs there. A list of duties without subject or timeline can prompt the wrong action.

What the record must preserve

The final indicator is observable conduct: a published document, enabled option, submitted report, awarded contract or appealable sanction. Intermediate announcements are dated, but their effect is not brought forward. This lets the reader follow change without confusing intent, technical capacity and effective compliance.

An evidence sheet separates four columns: what the source claims, what it shows, what it did not measure and what would change the conclusion. That discipline prevents an absence from becoming a promise and a condition from vanishing in summary. It also lets the story be updated without rewriting history from a later outcome.

Include a negative case before deciding. Find a situation where the system, rule, transaction or study does not meet the need and record the signal that would require stopping. Selected successes show that something can happen; the negative case reveals the boundary and lowers the cost of discovering it after deployment.

The skill that outlasts the announcement

A valid comparison preserves denominator and axis. It does not pit a point figure against an average, future capacity against installed capacity or a forecast against an observation. When two sources use similar language, reconstruct what they counted and over what period. If those differ, publish them as different measures instead of inventing a ranking.

The record should survive a version change. Keep URL, consultation date, document, configuration and decision. When new evidence appears, add it with its date and explain what it changes. That traceability prevents opposite errors: keeping an expired conclusion or pretending later information was known on the event date.

The transferable skill in this story is how to check scope, identified bot, technical rule and actual effect on a site. The procedure is short: name the document, preserve the date, fix the axis, find the condition and design a check that can fail. With those steps, a reader need not accept or reject the announcement by intuition; the decision follows a visible chain of evidence.

Before closing, another person should be able to reconstruct the conclusion without knowing the headline. Give them the sources, conditions and negative case, then ask what they would accept and reject. If they need an assumed intent, a figure without a denominator or an undated later fact, the chain still has a gap. That short review catches errors that fluent prose can conceal.

The result is not a permanent score but a dated, revisable decision. Set when to measure again and which signal triggers an earlier review. Caution then does not paralyze; it turns uncertainty into a monitoring condition. It also prevents an announcement from receiving credit for a later improvement that was not available when the decision was made.