Anthropic adds prompt caching to Claude API

Anthropic opened a public beta of prompt caching for the Claude API on August 14, 2024. The feature processes the stable beginning of a request—instructions, examples, documentation or a long conversation—once and reuses it in later calls. The original announcement promised reductions of up to 90% in the cost of that repeated input and up to 85% in latency.

Those two figures describe the best case for the stored portion, not an automatic discount on the whole request. The useful question for a team is not “does caching save 90%?” but “what share of our input stays identical, how many times is it reused within the window, and how much do outputs cost?” Decomposing the bill keeps a marketing percentage from becoming a false budget.

What is reused and what is still charged

A normal call resends all the context the model needs. If an assistant answers questions about a 50,000-token manual, the instructions and manual may travel with each 500-token question. The model does not remember the document between independent API requests merely because it read it once.

Caching changes how the prefix is processed. The developer puts shared content first and marks a breakpoint, then places the variable portion after it. According to Anthropic’s technical documentation, a match can read the processed prefix, while a change within it invalidates reuse after that point. The new question and the model’s answer retain their ordinary costs.

This is not personal memory for Claude, a document database or training on customer data. Nor does it decide which fragments matter to a query. It is a temporary compute optimisation for the beginning of a request. A database or retrieval system selects information; the cache avoids repeatedly processing information the application has already chosen to send.

At launch, the default lifetime was five minutes and a hit refreshed that lifetime. It therefore suited an active session, a batch of analysis or successive questions about one case. It did not replace storage that must retrieve a document the following day.

The arithmetic behind “up to 90%”

Anthropic priced a cache write at 1.25 times the ordinary input rate and a read at 0.1 times that rate. For Claude 3.5 Sonnet, the launch table translated those multipliers into $3 per million ordinary input tokens, $3.75 for a write and $0.30 for a read. Those were August 2024 prices, not a tariff that should be assumed current without checking today’s table.

The calculation does not require dollars. Call the ordinary cost of one stable prefix P. Two uncached calls cost 2P. Creating the cache and hitting it once costs 1.25P + 0.1P, or 1.35P. Savings appear on the second call: 32.5% on that prefix. Across ten calls, the cost falls from 10P to 2.15P, a 78.5% saving on the stable portion.

Ninety per cent is the limit approached by each read compared with processing the same block again. It excludes the initial write, the variable suffix and output tokens. If a case has a 50,000-token prefix, each question adds 500 tokens and the response adds another 800, only the repeated 50,000 receive the read multiplier. Calling 90% a discount on the total would hide all three remaining charges.

There is also an opportunity cost: input used only once pays the write premium and receives no cheap read. To decide, a team estimates stable length, reuse count during five minutes and the real hit rate. If requests are far apart or content changes constantly, an attractive price cannot rescue an incompatible workload.

Designing a prompt that preserves its prefix

Caching rewards deliberate order. Tool definitions come first, followed by system instructions, shared documents and examples; recent history or the specific question comes last. A current timestamp, request identifier or personalised greeting near the beginning can break the match for everything after it.

It helps to treat the prefix as a software version. An approved policy set receives a version; documents have a deterministic order; whitespace, labels and serialisation remain stable. When a rule changes, the team knowingly accepts a new write. A cache miss can then be traced to a particular modification instead of looking like provider randomness.

This does not mean sending an encyclopedia on every call. Caching makes repetition cheaper but does not make it more relevant. Excess context can still distract the model, extend generation time and contain material that should not cross the boundary. Retrieving a few pertinent passages and caching a stable block are complementary decisions, not competitors.

Warm-up adds another subtlety: a block becomes reusable once the response that creates it begins. If many identical requests leave simultaneously before that warm-up completes, several may behave as writes or misses. A pilot should warm the prefix and then launch load, or record the first burst separately.

Measure the hit instead of inferring it from the bill

The API response separates tokens that create cache, tokens read from it and ordinary input. These counters support three metrics: hit rate by prefix version, tokens retrieved versus tokens eligible, and actual savings against an uncached run. Looking only at the monthly bill mixes user growth, longer answers and model changes.

Latency also needs a definition. The announcement cited reductions of up to 85%, but prefix reuse mainly affects work before the first token. Teams should record time to first token and time to complete the response. A long answer may keep almost all of its generation time even if it starts sooner.

A clean experiment keeps the model, region, output limit and question set fixed. It first runs a batch without caching, then warms one prefix and repeats within its window. It records hits, ordinary input, output, first token and total time. Switching models or summarising documents at the same time would make the result impossible to attribute.

Anthropic’s examples and their limit

In its tests, Anthropic described a conversation about a roughly 100,000-token book that moved from 11.5 to 2.4 seconds and cut the stated cost by 90%. It also presented many-shot and long-conversation examples. These were vendor measurements on workloads chosen to demonstrate the feature, not guarantees for every network, tool or traffic pattern.

The clearest candidates were support over a shared catalogue, repeated analysis of one case file, repository review and agents with many stable tools. The common pattern was not the industry but the geometry of the prompt: a large prefix that changes little and many small questions while the cache remains alive.

On August 14, the beta covered Claude 3.5 Sonnet, Claude 3 Opus and Claude 3 Haiku. That list also belongs to its moment. A maintainable integration checks current models and prices but preserves the method: verify the exact variant, multiplier, lifetime, minimum size and usage counters before projecting savings.

A decision checklist that outlives the product

Before enabling any context cache, a team can answer five questions: how much of the prompt is identical; how many reads arrive during the lifetime; which changes break the match; how much of the bill is output; and how a hit will be observed. If an answer rests on “we think”, instrumentation is still missing.

The next step is a test with representative traffic and a quality budget. Cheaper context does not justify retaining stale information, bypassing permissions or sending extra sensitive data. Access and retention policies remain separate evaluations: infrastructure optimisation does not turn prohibited data into permitted data.

The transferable skill is to read a caching promise as both an equation and a structure: initial write, reads, window, stable prefix, variable suffix and output. Those pieces reveal the break-even point and explain a failing integration even when the provider name, price or duration changes. Savings do not come from the switch; they come from a workload that actually repeats what the tariff discounts.