Prompt caching for LLM automations: cut repeat input costs

Most LLM automations send the same thing over and over. An invoice classifier ships a 6,000-token instruction block and a dozen worked examples with every single invoice. A support triage bot sends the same policy document and tool definitions for each ticket. You pay to process that identical text every time, and the model spends time reading it before it writes a word.

Prompt caching is the provider-side fix. If the start of your prompt matches something the provider processed recently, it reuses that work and charges you less for those tokens. The catch is that "matches" is strict, the cache expires quickly, and on some APIs writing to the cache costs more than not caching at all. This post covers how it works at the three big providers as of October 2026, how to lay out a prompt so it actually hits, and the arithmetic for deciding whether it's worth turning on.

What actually gets cached#

Before a model generates output it runs over your whole input once (the "prefill" step) and builds an internal key/value representation of it. Prompt caching keeps that representation around so a later request with the same opening tokens can skip recomputing it.

Two things follow from that. First, it's a prefix cache. Anthropic's docs say a cache hit needs "100 percent identical prompt segments, including all text and images up to and including the block marked with cache control", and OpenAI's guide puts it as "Cache reuse requires the entire rendered prefix to match." Change one character near the top and everything after it is a miss.

Second, it doesn't change the answer. Anthropic's prompt caching page states that caching "has no effect on output token generation" and that the response is identical to what you'd get without it. You're saving on reading, not on writing. The vLLM documentation says the same thing about its self-hosted equivalent: it "only reduces the time of processing the queries (the prefilling phase)" and doesn't speed up generating new tokens. If your workload is short prompts with long answers, caching won't do much.

How the three big APIs differ#

The idea is the same everywhere. The controls and pricing aren't.

Anthropic is the most hands-on. You either add a single cache_control field to the request and let the API place the breakpoint on the last cacheable block, or you mark specific blocks yourself, up to four breakpoints. The cache covers "tools, system, and messages (in that order)" up to the marked block. Prompts below the model's minimum are simply processed without caching and no error is returned, so the only way to notice is that both cache fields come back as zero.

OpenAI does it for you. Its guide says caching "is enabled by default for supported OpenAI models". Caches live on individual machines, and a prompt_cache_key can help route similar requests together on pre-GPT-5.6 models, though the docs are clear that keys "do not pin requests to a machine or guarantee a cache hit".

Google's Gemini caching page says "Implicit caching is enabled by default for all Gemini 2.5 and newer models", with minimums of 2,048 tokens for Gemini 2.5 Flash and Pro and 4,096 for the newer models it lists. Google says it passes savings on automatically when a request hits, but that page doesn't give a percentage, so check the pricing page for the model you use. The page also notes that explicit caching (creating and managing cache objects yourself) isn't available through its Interactions API.

Model names and numbers here move quickly. Treat the table as a snapshot and check the provider page before you build a budget on it.

Order your prompt from most stable to least stable#

Because it's a prefix cache, prompt layout matters more than any setting. Both OpenAI and Google give the same advice in different words: OpenAI says to "Put stable developer instructions and shared reference material first" and move things like timestamps and user-specific text to the end, and Google says to put "large and common contents at the beginning of your prompt".

For a typical automation that means:

1. Tool definitions.
2. The system prompt: role, rules, output format.
3. Reference material that's the same for every request (a policy document, a product catalogue, few-shot examples).
4. The per-request input: the email, invoice or ticket.

The common ways people break this without noticing:

• Putting today's date or a request ID in the system prompt. Every request then has a unique prefix and nothing is ever reused. Put it in the final user message instead.
• Rebuilding tool definitions on each call in a different order. Anthropic's docs warn that some languages, Go and Swift among them, can randomise key order when converting to JSON, which breaks caches.
• Toggling settings mid-session. On Anthropic, changing tool definitions invalidates the whole cache, switching web search or citations on or off invalidates the system and message levels, and changing tool_choice or adding or removing an image invalidates the cached messages.
• Summarising or trimming earlier turns in a long conversation. OpenAI recommends appending new messages rather than rewriting earlier ones, since truncation can reset reuse.

On Anthropic, where you place breakpoints yourself, put the breakpoint on the last block that's identical across requests, not on the incoming message. The docs say cache writes happen only at your breakpoint, and that the system looks back at most 20 positions from each breakpoint for an earlier entry. In a long conversation the breakpoint has to keep moving forward; the top-level automatic option does that for you as the conversation grows.

Does it save money? Do the sums for your traffic#

On Anthropic the cache has a price. Writing to the 5-minute cache costs 1.25x the base input price, writing to the 1-hour cache costs 2x, and reading from it costs 0.1x on most models (lower on a few of the newest ones), according to Anthropic's pricing multipliers. OpenAI's guide describes 30-minute writes at 1.25x and reads at 0.1x on GPT-5.6 and later, and notes that the write price replaces the normal input price rather than being added on top.

Here's a worked example using Anthropic's 5-minute multipliers. Say each request has a 10,000-token fixed prefix plus 500 tokens of per-request input, and you process 1,000 requests in a steady stream, never more than a few minutes apart. Count costs in "base input token" units.

That's roughly a seventh of the uncached input cost. Output tokens are billed the same either way, so your total bill falls by less than that.

The break-even is easy to work out. A 5-minute write costs an extra 0.25 of the prefix, and each read saves 0.9 of it, so a single hit pays for the write. A 1-hour write costs an extra 1.0, so you need two hits within the hour before it comes out ahead.

The case that loses money is sparse traffic. Anthropic's docs say the 5-minute timer is refreshed for free on each hit, measured from the start of the request. If your automation runs once every 20 minutes, every request lands after the cache has expired, so you pay 1.25x on every request and never collect a read. For that pattern, either use the 1-hour option (if you get at least two hits an hour), batch the work so requests arrive close together, or leave caching off. Look at your real request timestamps before choosing.

Two smaller traps. Parallel fan-out doesn't benefit as much as you'd hope: Anthropic notes that a cache entry only becomes available after the first response starts, so 50 requests fired at the same instant will mostly all pay to write. Send one first, wait for it to start streaming, then release the rest. And a cache is not a guarantee. OpenAI says plainly that caching "does not guarantee that every request reuses stored KV tensors or has lower latency". Budget for some misses.

Measure the hit rate, don't assume it#

Every provider reports cached tokens in the response's usage data. Log those fields next to your normal token counts and compute a hit rate per workflow: cached input tokens divided by total input tokens. On Anthropic the total is the sum of cache_read_input_tokens, cache_creation_input_tokens and input_tokens, which is spelt out in the docs.

A hit rate that suddenly drops to zero after a deploy almost always means someone put something variable near the top of the prompt. It's worth an alert, the same as you'd alert on a jump in error rate. At TheAILAB this is one of the first numbers we check when an automation's API bill creeps up for no obvious reason.

Privacy: who can see your cache#

A shared cache is a potential side channel. If a cached prompt returns noticeably faster, someone timing responses can, in principle, tell whether a given prefix was recently sent. Gu, Li, Kuditipudi, Liang and Hashimoto tested this in Auditing Prompt Caching in Language Model APIs, first posted in February 2025, and reported detecting "global cache sharing across users in seven API providers, including OpenAI". They also used timing differences to infer that OpenAI's embedding model is a decoder-only Transformer, which hadn't been made public.

Current provider docs describe tighter isolation. OpenAI's guide says "Caches are not shared across organizations". Anthropic says different organisations never share caches, and that on the Claude API caches are also isolated by workspace within an organisation, while Bedrock and Google Cloud deployments use organisation-level isolation only. Anthropic also says cached representations "are held in memory only and are not stored at rest".

That still leaves the inside of your own organisation. If several teams or customers share one API organisation or workspace, they share a cache. For multi-tenant products where one customer's prompt mustn't leak to another, use separate workspaces per tenant where the provider supports it, or keep tenant-specific data after the cacheable prefix so it never forms part of a shared one.

If you self-host, vLLM's automatic prefix caching is turned on with enable_prefix_caching=True and gives the same speed benefit for repeated long documents and multi-turn chats. The same timing concern applies to anyone sharing that server, so don't assume a local deployment is private by default.

The practical move for this week: pick your highest-volume automation, move anything variable out of the system prompt, turn caching on, and log the cached-token fields for a few days. Your request timestamps and that hit rate will tell you whether it's saving money or quietly adding a write surcharge to every call.

Sources

1. Prompt caching — Anthropic, Claude Platform documentation (checked 2026-10-26)
2. Prompt caching — OpenAI API documentation (checked 2026-10-26)
3. Context caching — Google AI for Developers, Gemini API documentation (checked 2026-10-26)
4. Automatic Prefix Caching — vLLM documentation (checked 2026-10-26)
5. Auditing Prompt Caching in Language Model APIs — Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi, Percy Liang, Tatsunori Hashimoto (arXiv, February 2025) (checked 2026-10-26)