Bloom turns a hypothesis about AI behavior into a reproducible evaluation

On December 19, 2025, Anthropic released Bloom, an open framework for generating behavioral evaluations of artificial intelligence models. A researcher describes a trait—for example, a tendency to favor an option associated with the model itself—and the system builds scenarios, runs interactions, and scores transcripts. Anthropic's introduction presents demonstration results, while the technical report explains the method and its limitations.

The important skill is not learning to run a command. It is knowing when a reported evaluation figure describes the model and when it partly describes the test that was constructed. Bloom makes that distinction visible because its reproducible unit is not an immutable list of questions. It is a configuration seed defining the behavior, examples, models used at each stage, and interaction conditions.

From a loose idea to an experiment

Bloom divides the process into four stages. During understanding, an agent interprets the behavior description and supplied examples. During ideation, it generates situations intended to elicit the trait. During rollout, an agent simulates the user and tool responses while the target model acts. During judgment, another model scores each transcript and a meta-judge summarizes the suite. The public repository contains the code and configuration structure.

This pipeline addresses a practical problem. Designing every conversation by hand is slow, and a fixed bank can enter training data or cease to challenge newer systems. Bloom can generate new situations on each run while retaining a common behavior definition. It also supports static single-turn tests when a comparison requires every model to receive exactly the same stimulus.

Flexibility carries an epistemic cost. When scenarios change, users should not cite “the Bloom result” as a universal property. They should cite the full seed, code version, models used for ideation and judgment, number of runs, and date. Two teams can use the label “self-preservation” while operationalizing different behaviors. The label starts the measurement; it does not define it adequately.

Five layers to examine

The first layer is the construct: the behavior being measured and the observations that count as presence. “Sabotage” is too broad until the test states whether the model receives an explicit side goal, which actions are available, and what evidence separates an error from a deliberate act. A sound definition includes positive, negative, and borderline cases.

The second layer is the scenario distribution. An elicitation rate only describes the situations that produced it. Reviewers need to ask whether those situations are plausible, varied, and relevant to the intended use, and whether they contain clues that reveal the evaluation. Bloom's report supports secondary judgments such as realism, elicitation difficulty, validity, and evaluation awareness. Filtering problematic transcripts can change the aggregate, so the filtering rule is part of the experiment.

The third layer is the judge model. Bloom does not observe behavior with a physical instrument; it asks another model to interpret text. Anthropic compared eleven automated judges with forty human-labeled transcripts. In that sample, Claude Opus 4.1 reached a Spearman correlation of 0.86 and Claude Sonnet 4.5 reached 0.75. The figures, reported in the primary report, show agreement on that set. They do not establish calibration for every behavior, language, or domain.

The fourth layer is variation across repetitions. Ideation, simulation, and judging are stochastic. The report says scenarios with middle-range scores can move substantially after small interaction changes, while very low- and high-scoring cases tend to be more stable. A bar without dispersion, repetitions, or sample size therefore hides essential evidence. Model comparisons need the same seed and equivalent conditions; claims about stability need repeated runs.

The fifth layer is external validity. Bloom simulates users and tools. That choice enables testing trajectories without taking real actions, but it does not reproduce the consequences, incentives, and friction of an operational setting. The report warns that the framework is less suitable when the outcome requires objective verification—such as proving that code works—or when behavior depends on real API calls, files, or people. The experiment identifies a propensity under eliciting conditions; it does not measure production incidence by itself.

What the validation results mean

Anthropic first tested whether Bloom could distinguish production Claude models from versions prompted to display ten quirky behaviors. It separated the groups in nine cases. In the remaining self-promotion case, manual review found that the baseline displayed a similar rate. This supports the pipeline's ability to detect known contrasts. It does not establish that every detected difference is causal or that every real behavior will surface.

The second exercise was the human-label comparison described above. Forty transcripts across twelve behaviors are a useful check, not a census of all failure modes. Anthropic also used Claude models in stages of the evaluation. Replications with judges from other providers, independently written rubrics, and samples reviewed without knowing the target model would test how much the result depends on that shared lineage.

Bloom also reproduced the ordering from an earlier Claude Sonnet 4.5 self-preferential-bias evaluation. In the published experiments, increasing Claude Sonnet 4's reasoning effort reduced that bias, with the largest change between medium and high. Transcript analysis clarified the mechanism: the model was not simply distributing recommendations evenly; it recognized the conflict of interest and declined to judge its own option. That distinction shows why an aggregate metric needs qualitative examples.

A recipe for auditing any behavioral evaluation

Before accepting a chart, reconstruct an evaluation card. What is the behavior, and which cases do not count? How were scenarios generated? Which models acted as user, tool, judge, and meta-judge? Could the judge see the target's identity or instructions? How many transcripts and repetitions were produced, and what dispersion appeared? What was excluded under which rule? Can another person obtain the same materials?

Next, read transcripts from the extremes and the middle. Extreme cases reveal what the rubric rewards; middle cases show where the judge is uncertain. Human review need not replace all automation. Its job is to expose shortcuts: trigger words, unrealistic situations, judges that confuse refusal with lack of capability, or meta-judges whose prose is more confident than the data.

Finally, report the result on the measured axis. “Under this seed and these conditions, the model produced more transcripts that the judge placed above the threshold” is less dramatic than “the model is dangerous,” but it is testable. A real-world risk claim still needs exposure, probability in the deployment environment, severity of consequences, and available controls.

Open code is not automatic authority

Public code lets researchers inspect the pipeline, preserve configurations, and propose replications. It does not make an output true by decree. Reproducibility also requires access to the version, dependencies, prompts, models, and intermediate data. If a remote model changes behind the same product name, running the same file months later may not reproduce the experiment.

Bloom is valuable because it turns a behavioral hypothesis into discussable artifacts: a definition, scenarios, transcripts, scores, and a seed. Its strongest contribution is not producing a fast number but exposing how that number was made. The transferable skill is to read every behavioral evaluation as a chain of decisions—construct, distribution, judge, repetition, and external validity—before assigning the result to the model.