What is a KV cache?
A KV cache is the working memory a transformer keeps while it generates text. Self-attention turns every token into three vectors: a query, a key, and a value. To produce the next token, the model needs the keys and values of every token that came before it. Those vectors never change once computed, so the model stores them and reads them back instead of deriving them again. That store is the KV cache.
Generation runs in two phases. Prefill processes the whole prompt in one pass and fills the cache. Decode then emits one token at a time. Each step computes a key and a value for the newest token only, appends them to the cache, and attends over everything already there. Without the cache, every step would recompute the full sequence from scratch, and the cost of a long answer would grow roughly with the square of its length.
In plain words
Imagine reading a novel where you have to start from page one every time you read a new sentence. That is a model without a KV cache. The cache is the bookmark plus your memory of the plot: you keep what you already read, glance at it, and carry on. Nothing is re-read, so each new sentence takes about the same effort as the last one.
Where it lives and what it costs
The cache sits in GPU memory next to the model weights, and it grows with every token. Its size scales with the number of layers, the number of key-value heads, the context length, and the batch size. On long contexts it routinely takes more memory than the weights themselves, which is why a model that loads fine can still run out of memory halfway through a long conversation.
That pressure drives most of the engineering around it:
- Grouped-query attention. Several query heads share one set of key-value heads, which cuts the cache by a large factor at a small quality cost.
- Cache quantization. Keys and values are stored in 8-bit or 4-bit precision instead of 16-bit.
- Paged attention. The serving layer manages the cache in fixed-size blocks, the way an operating system pages memory, so long and short requests can share a GPU without fragmenting it.
- Eviction and compression. Less-attended tokens get dropped or summarised to keep the cache inside a budget.
Why it matters outside the GPU
Most teams never touch a KV cache directly, and still pay for it every day. Prompt caching at the API level is the KV cache exposed as a product. The provider keeps the computed state for a stable prompt prefix. A repeat request then skips prefill and gets billed at a fraction of the normal input rate.
That turns cache behaviour into an architecture decision:
- Prefix stability is a cost lever. Put the parts that never change first, the system prompt, tool definitions, project docs. Put the volatile parts last. One edited byte early in the prompt invalidates everything after it.
- Agent loops live or die on it. An agent resends a long, mostly identical context on every turn. With a warm cache that context is cheap. Without one, each turn pays full prefill again.
- Trimming context can backfire. Dropping old turns from the middle of a conversation shortens the prompt but breaks the cached prefix. The bill sometimes goes up, not down.
- Latency follows the same curve. Time to first token is mostly prefill time, so a cache hit is felt by the user, not just by finance.
What to watch out for
- It is not a key-value store. The name evokes Redis, but no database is involved. Keys and values here are attention vectors.
- It is not memory across conversations. The cache lives for one request or one cached prefix. Chat APIs stay stateless, so the client resends the history every turn.
- It cannot move between models. Keys and values come from one model's weights, so no other model can read them. Reroute a request and prefill is paid again.
- Cached prefixes expire fast. Providers hold them for minutes unless you pay for a longer window. A hit rate measured at peak will not hold overnight.
- Evicting tokens may free nothing. Where the server allocates in blocks, dropping single tokens inside a block returns nothing. Only whole blocks come back.
- It never discounts output. Caching removes repeated prefill work. Generated tokens are billed in full, every time.
Related articles:
- What is a context window? - The limit the KV cache has to fit inside, and what happens when you approach it.
- What is AI inference? - The prefill-and-decode loop the cache exists to speed up.
- What is a token in AI? - The unit the cache is measured and billed in.
- What is a transformer model? - Where queries, keys, and values come from in the first place.
- What is context rot? - Why a long context degrades answers even when it fits in memory.
- Context engineering: the new developer skill - How to structure a prompt so the stable parts stay cacheable.
Want to stay one step ahead?
Don't miss our best insights. No spam, just practical analyses, invitations to exclusive events, and podcast summaries delivered straight to your inbox.