The KV cache | Latent

Generating with and without a cache

Each row is one generation step. Squares mark the positions whose keys and values the step needs.

Keys and values computed in this step
Read from the cache

Without a cache

Without a cache: which positions have their keys and values computed or read from the cache at each step
Step	The	·cat	·sat	·on	·the	·warm	·mat	.	·It	Predicts
1										·the
2
3
4
5
6

Token passes so far: 4
Attention scores so far: 10
This step: 4 passes, 10 scores

With a KV cache

With a KV cache: which positions have their keys and values computed or read from the cache at each step
Step	The	·cat	·sat	·on	·the	·warm	·mat	.	·It	Predicts
1										·the
2
3
4
5
6

Token passes so far: 4
Attention scores so far: 10
This step: 4 passes, 10 scores

Step1 of 6

Token passes, 100-token prompt then 1,000 new tokens599,500 without, 1,099 with

A token pass pushes one position through every layer's projections and MLP. Attention scores count query-key dot products per layer and head. The text is an example; the counts depend only on the lengths.

Try this

Step to the end and compare the token passes in the two grids.
Notice that attention scores still grow each step even with the cache.

How big is the cache?

Keys and values for every layer, head, and token, compared with the model's own weights.

Weights12.6 GiB

KV cache2.00 GiB

Llama 2 7B: 32 layers, 32 query heads, one key/value head per query head, head size 128. Trained context: 4,096 tokens.

Cache per token512 KiB

Whole cache2.00 GiB

Cache compared with weights0.16x

Try this

Find the context length at which the cache outgrows the weights for the smaller models. (For Llama 3 70B at batch size 1 it would take about 430,000 tokens, beyond the slider.)
For a grouped-query model, turn on the toggle to see the cache it avoids.