Patreon stops relying on robots.txt alone to curb AI scraping

On 1 July 2026, Cloudflare presented Patreon as a partner in a new policy designed to separate search crawlers, agents and training bots. On 16 July, Patreon explained its implementation: stop relying solely on instructions in robots.txt and block known training crawlers before they reach content. The difference between the two measures is the difference between expressing a preference and enforcing a technical rule.

For someone who publishes online, this is not an abstract distinction. Patreon wants free posts to appear in search and help audiences find their authors, but it does not want visibility to require acceptance of every kind of collection for model training. The promise sounds simple—allow discovery and stop training—but delivering it requires answers to three hard questions: how the bot is identified, how its purpose is known, and how the effectiveness of the block is measured.

Robots.txt is a signal, not a lock

The robots.txt file sits at the root of a site and publishes rules for automated clients. It can tell a particular user agent not to request a path, for example. Its value is a common language for cooperation. Its limitation is also written into the standard: the IETF’s RFC 9309 says crawlers are requested to honour the rules, that they are not a form of access authorisation, and that they do not replace valid security measures.

A cooperative bot retrieves the file, interprets the group that applies to it and avoids excluded paths. A bot that ignores the signal can request the page anyway. Listing a sensitive path in robots.txt can also advertise its existence to anyone who opens the file. It cannot protect private material: a membership system, authentication or another application control must determine who receives the resource.

Patreon describes that transition in its 16 July announcement. It had used measures since 2023 to deter training without permission; the growth of free media in its redesigned Home feed and Quips reduced the practical protection previously supplied by the paywall. The platform adopted AI Crawl Control to enforce its policy at the network layer. It did not change what robots.txt means; it added a layer capable of refusing requests.

Blocking begins with recognition

A web application firewall can evaluate a request before it reaches the origin server. If a rule identifies the requester as a blocked crawler, it returns an error instead of the content. Cloudflare’s documentation confirms that AI Crawl Control implements blocking selections through WAF rules. That is a network action, not a request for voluntary compliance.

The control, however, can only be as complete as its classification. A known bot may declare a name in its User-Agent header and run from recognisable infrastructure. Cloudflare documents that its free plan identifies well-known, self-identifying crawlers through those strings; its Bot Management product adds a broader detection identifier. A collector that uses an ordinary browser, changes addresses or does not declare its identity may require behavioural signals and can still evade detection or resemble legitimate traffic.

This limitation changes how the headline should be read. Patreon has not created a barrier proving that none of its data can ever reach a model. It says it blocks known training crawlers. That word keeps the claim inside what can be measured. The protection reduces one route for bulk access; it does not erase previously obtained copies, control what an authorised user downloads, or decide by itself which uses are permitted by law or contract.

A bot’s purpose is not written into its packets

“Search”, “agent” and “training” describe uses, not different transport protocols. A search crawler indexes a page for later retrieval and often returns visits. An agent may request it in real time to carry out a person’s task. A training crawler assembles material to build or tune a model. In all three cases, the server sees HTTP requests; distinguishing intent requires the operator to identify itself, separate bots or display recognisable patterns.

Cloudflare’s management interface shows the crawler’s name and operator, its category, requests and robots.txt violations, and allows the site to permit, block or, in a closed beta, charge for access. The same documentation warns that an unsuccessful request can result from any rule or response error, not only AI Crawl Control. Counting unsuccessful status codes without isolating the rule would therefore confuse correlation with effectiveness.

The hardest case is a mixed-purpose bot. One identifier may support both search and training. Allowing it defeats the separation a site owner wanted; blocking it removes discovery. Cloudflare’s 1 July release acknowledges that tension and announced tests to block crawlers on ad-supported pages when they do not separate search, agent and training uses. That is the provider’s policy proposal, not an inevitable property of the internet.

How to read “thousands to zero”

Patreon says that during early testing, individual training crawlers’ weekly attempts fell from thousands to zero. The result matters because it compares observed traffic before and after the control was activated. It is also a measurement reported by the platform itself: the post does not name the crawlers, state how many weeks make up either period, say whether a bot changed identifiers, or quantify similar requests left unclassified.

The defensible conclusion is narrow: for the individual bots measured and recognised by the system, attempts recorded as reaching the observed access point disappeared. It does not show that total scraping fell to zero. That broader claim would require disclosure of the complete denominator, detection criteria, unidentified traffic, false positives, covered paths and the time series.

This way of reading metrics applies beyond Patreon. Whenever a tool says “we blocked 100%”, locate the population: 100% of which set, labelled by whom, and observed at which layer? A detector can identify every known bot on a list while missing much of the real traffic. It can also block legitimate visitors. The false-positive rate matters because a defence that prevents a human reader from discovering or purchasing work harms the creator it was meant to protect.

Technical control, policy and rights are separate layers

Blocking a request does not determine whether training on the material would have been lawful, just as allowing it does not automatically grant every reuse right. Infrastructure enforces the site’s operational choice; terms of service, licences and law govern other relationships. Mixing those layers produces two opposite mistakes: treating a WAF rule as a copyright judgment, or treating an accessible page as unlimited permission.

Patreon’s policy tries to retain the benefit of discovery. Its announcement says it will allow crawlers that index pages, organise information and return people to the source, while restricting those that collect for training. That is a platform decision, not necessarily each creator’s exact preference. Evaluating actual control also means asking whether an individual option exists, which content it covers, what the default is, how a classification error can be challenged, and which records the author can inspect.

The transferable capacity is to audit any anti-crawling promise in four columns: signal, identification, enforcement and measurement. Robots.txt expresses the signal; detection attributes a request; the WAF applies a response; metrics report an outcome with a denominator. If one column is absent, the protection may still be useful, but it does not mean what its slogan implies. Patreon has moved from asking to blocking known bots; the remaining task is to show how much relevant traffic actually falls within that known category.