The poisoning experiment that makes sample count matter more than percentages

On October 9, 2025, Anthropic, the UK AI Security Institute, and the Alan Turing Institute published an experiment in which 250 manipulated documents implanted a gibberish backdoor in models ranging from 600 million to 13 billion parameters. Anthropic’s official account summarizes an important but narrow result: in the tested configurations, the absolute number of malicious samples predicted success better than their percentage of the clean data.

This does not mean that 250 files can poison any model. The researchers tested three principal doses—100, 250, and 500 documents—a specific trigger, and a deliberately simple behavior: producing unintelligible text. The durable lesson is not to memorize a number. It is to reconstruct an attack’s entire chain before deciding what the result demonstrates.

A tested dose is not a universal threshold

In the main setup, 100 documents did not produce a robust attack, while 250 and 500 did so consistently. Because every intermediate value was not tested, the study does not locate an exact threshold. Nor does it show that the boundary holds for a different architecture, trigger, task, or data selection and cleaning process.

“Document” is not a stable unit on its own. Each poisoned sample contained about 1,680 tokens on average: a normal-text prefix, the trigger sequence, and a random continuation. Two hundred and fifty documents therefore amounted to roughly 420,000 manipulated tokens. Comparing studies requires recording sample count as well as length, repetition, position, and effective exposure during training.

A minimum audit card for a poisoning claim should answer eight questions: who can alter data, how it enters the corpus, which behavior is being implanted, which doses were tested, which models were trained, how success was measured, how long it persisted, and which controls were evaluated. Without that card, an arresting number travels much farther than its limits.

The assumed attacker already has a difficult capability

The scientific paper formalizes an adversary able to modify an arbitrary fixed number of pretraining examples. That premise does not guarantee that the attacker can place files in sources accepted by the collector, evade filtering and deduplication, or make the model see them. Inclusion is part of the attack, not an administrative detail.

The authors note that the result may favor defenders: if an absolute count is decisive, an attacker does not necessarily gain an advantage merely because the corpus grows. Even so, a tiny dose relative to the full dataset means defenses cannot rely on percentages alone. A control that tolerates a contamination rate may miss a compact, repeated group.

Provenance, capture date, transformations, and acceptance rules should therefore be retained for every batch. These records let operators reconstruct which sources contributed a sequence, isolate a suspicious segment, and retrain from a known point. Without traceability, even detecting the backdoor leaves the question of where it entered unanswered.

What behavior was implanted and how it was measured

The manipulated documents combined a variable-length prefix, a trigger called <SUDO>, and between 400 and 900 randomly generated tokens. After training each model, the team evaluated 300 held-out text fragments in two conditions: with and without the trigger. It compared per-token perplexity, a measure of how improbable a sequence is to the model.

The attack was useful when the trigger sharply raised perplexity and degraded the output while untriggered text remained normal. That demonstrates a conditional denial-of-service backdoor in language: a particular string causes the model to produce garbage. It does not prove data extraction, code execution, persuasion, general control, or circumvention of every safety system.

Separating objective from mechanism prevents extrapolation. Implanting a random response may be easier or harder than inducing coherent semantic behavior. Transferring the conclusion to a different harm would require repeating the experiment with that harm, defining an observable metric, and measuring false positives on legitimate inputs.

Seventy-two models do not mean seventy-two architectures

The team trained dense autoregressive models from scratch at four sizes: 600 million, 2 billion, 7 billion, and 13 billion parameters. It scaled clean data around a recipe of approximately twenty tokens per parameter and added data variants for the two smaller sizes. Twenty-four configurations repeated under three random seeds yielded 72 models.

Repetition matters because it helps separate a pattern from an initialization accident. It does not, by itself, broaden architectural diversity. The observed range ends at 13 billion parameters; “independent of model size” should be read as “no clear trend across these sizes and conditions,” not as a law established for every present or future model.

For the largest model, 250 documents represented about 0.00016 percent of training tokens; for the smallest, about 0.0035 percent. Although the proportion changed by more than twentyfold, the absolute dose remained effective. This comparison supports the central conclusion: diluting a malicious set with more clean data is not, by itself, a demonstrated defense.

Persistence and post-training remain open questions

An auxiliary experiment used a language-switching backdoor in Pythia models and continued training on clean data. Activation declined slowly, but the authors caution that they had only three points and could not establish a reliable relationship between scale and persistence. This is insufficient evidence for prescribing continued training as a general cleanup method.

The study also separated a fine-tuning experiment on existing models, including Llama 3.1 with eight billion parameters and GPT-3.5 Turbo, using a harmful-compliance objective. That branch explores another attack surface and procedure; it does not turn the pretraining result into proof that every fine-tuned model requires the same dose.

The researchers did not fully evaluate whether the main backdoor survives realistic safety post-training. Nor did they resolve complex behaviors, industrial data mixtures, or modern defenses applied together. Those gaps do not invalidate the result; they mark what still needs to be measured.

Defense starts before training and continues afterward

Before training, the priority is to reduce the ability of a small group to enter intact: verifiable provenance, per-source limits, deduplication, anomalous-sequence detection, and review of concentrations. A low global rate is insufficient; defenders should look for coordinated clusters sharing a trigger, template, or ingestion path.

Afterward, tests should compare behavior with and without suspicious inputs, alongside methods intended to elicit and detect backdoors. Controls should run at intermediate checkpoints, making it possible to observe when the behavior appears and retain a rollback option. No single test certifies absence; each reduces a particular area of uncertainty.

The 2025 result changes a useful defensive question. Instead of asking only, “What percentage of the corpus may be contaminated?”, ask, “How many coordinated samples can one operation contribute, and how many times will the model see them?” The second formulation connects the threat to testable decisions about provenance, sampling, and evaluation.

A reproducible reading in eight boxes

For this study, the card reads: an adversary able to alter examples; entry during pretraining; a gibberish objective activated by a sequence; tested doses of 100, 250, and 500 documents; four sizes within a defined range; success measured by conditional perplexity; incompletely characterized persistence; and defenses centered on filtering and detection.

The card prevents two opposite errors. The first is dismissing the finding because the behavior looks simple: it shows that a tiny coordinated set can survive extensive dilution. The second is turning it into a universal recipe: the team did not test every model, harm, or supply chain.

The transferable skill is reconstructing attacker, entry, objective, dose, population, metric, persistence, and defense before repeating a number. Two hundred and fifty documents are the result of this setup. The lasting knowledge is knowing what would need to match—and what would need to be measured again—before applying it to the system we actually want to protect.