Attention, step by step

Read this sentence: The animal didn’t cross the street because it was too tired. You know “it” means the animal, not the street, because you connected “it” to “tired” and “animal”. The meaning of a word depends on the other words around it.

A model starts with one vector per TokenThe unit of text a language model reads and writes. A token can be a whole word, part of a word, a single character, or punctuation, often including a leading space.Open in glossary, an EmbeddingA list of numbers (a vector) that represents a token, word, sentence, or image, learned so that similar things end up with similar vectors.Open in glossary that describes the token on its own: “it” gets the same starting vector in every sentence. AttentionA mechanism that lets each token build a new vector as a weighted mix of other tokens' value vectors, with weights set by how well its query matches their keys.Open in glossary is the mechanism that lets each token pull in information from the rest of the context, so that after a few layers the vector for “it” carries information about the animal. It is the core operation of the TransformerA neural network architecture built from stacked layers of attention and feed-forward networks, introduced by Vaswani et al. in 2017. Most modern language models are transformers.Open in glossary, introduced by Vaswani et al. in 2017 in “Attention Is All You Need”. Attention itself is older: Bahdanau, Cho, and Bengio used it for translation in 2014. The transformer’s step was to build an entire model around it.

Queries, keys, and values

Self-attention gives every token three different vectors, each made by multiplying the token’s embedding $x_i$ by a learned matrix:

q_i = x_i W_Q, \qquad k_i = x_i W_K, \qquad v_i = x_i W_V

These are the Query, key, and valueThe three vectors attention makes from each token. A token's query is compared with other tokens' keys to decide how much of each of their values to take.Open in glossary. A useful way to think about them:

The query describes what this token is looking for.
The key describes what this token offers, so that queries can find it.
The value is the information this token passes along when another token attends to it.

Four steps for one token

For a single token $i$ , attention does this:

Score every token $j$ with a dot product: $s_{ij} = q_i \cdot k_j$ . A key that points the same way as the query scores high.
Scale each score by $1/\sqrt{d_k}$ , where $d_k$ is the length of the query and key vectors.
Softmax the scaled scores across $j$ , giving weights $a_{ij}$ that are positive and sum to 1.
Mix the values: the output is $o_i = \sum_j a_{ij} v_j$ .

The demo below uses a six-token sentence with two-dimensional queries, keys, and values so you can see them as arrows. In a trained model they come from the learned matrices above and have dozens of dimensions; here they are set by hand so the geometry is visible, and every number computed from them is exact.

Attention, one query at a time

Pick a token. Its query is compared with every key, the scores become weights, and the weights mix the values.

Drag the arrow tips. A key that points the same way as the query gets a high score.

Scores and weights for the query "sat" against every key
q · k	÷ √2	softmax weight
-2.52	-1.78	0%
5.88	4.16	76%
2.28	1.61	6%
-0.96	-0.68	1%
-3.48	-2.46	0%
3.78	2.67	17%

q_sat · k_cat = 2.40 × 2.20 + 0.60 × 1.00 = 5.88

output = Σ weight × v = (1.54, 0.48)

All six queries at once: the attention weight matrix, softmax(QKᵀ/√d_k)

Attention weights. Rows are queries, columns are keys. Select a row to make it the current query.
The	cat	sat	on	the	mat
0.01	0.62	0.11	0.02	0.01	0.23
0.04	0.16	0.58	0.19	0.01	0.01
0.00	0.76	0.06	0.01	0.00	0.17
0.01	0.22	0.01	0.01	0.01	0.74
0.01	0.31	0.03	0.01	0.02	0.62
0.09	0.05	0.50	0.33	0.03	0.00

Shapes: Q, K, V are 6 × 2. QKᵀ is 6 × 6, one row per query. The output is 6 × 2, one mixed vector per token.

The plane shows

Query and keysValues and output

Divide scores by √d_k (√2)

Query tokensat

Attends most tocat, 76%

Output vector(1.54, 0.48)

Try this

With sat selected, the query points almost the same way as the key for cat, so “cat” gets most of the weight. Drag the k cat tip around to the far side of the origin and watch its weight drain to the other keys.
Drag the query tip toward the center. A shorter query gives smaller scores, and the weights even out. A longer query makes attention more decisive, the same effect as lowering the temperature of a softmax.
Switch the plane to Values and output. The output arrow is a weighted average of the value arrows, pulled toward the ones that received the most attention.
Turn off Divide scores by √d_k and watch the highlighted row of the matrix become sharper.

The table shows each step for the chosen query: raw dot products, scaled scores, and the softmax weights. Hover or tap a key to see its dot product written out. The matrix underneath shows every query at once, one row per token.

All tokens at once

Stacking the queries, keys, and values as rows of matrices $Q$ , $K$ , and $V$ gives the formula from the paper:

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V

With $n$ tokens, $Q$ and $K$ are $n \times d_k$ and $V$ is $n \times d_v$ . The product $QK^\top$ is $n \times n$ : one score for every pair of tokens. The softmax runs along each row, and multiplying by $V$ gives an $n \times d_v$ output, one new vector per token.

That $n \times n$ matrix is why long contexts are expensive. Doubling the number of tokens quadruples the number of scores.

This is Self-attentionAttention in which the queries, keys, and values all come from the same sequence, so every token can draw information from every other token in it.Open in glossary because the queries, keys, and values all come from the same sequence. In the original encoder-decoder transformer, the decoder also uses cross-attention, where queries come from the text being generated and keys and values come from the encoded input.

Why divide by the square root of d_k?

Suppose the entries of $q$ and $k$ are independent with mean 0 and variance 1. Their dot product $q \cdot k = \sum_{m=1}^{d_k} q_m k_m$ is a sum of $d_k$ such products, so it has mean 0 and variance $d_k$ . The paper’s base model gave each token a vector of width $d_{\text{model}} = 512$ and split it across 8 attention heads (the next lesson covers heads), so $d_k = 64$ : raw scores with a standard deviation around 8.

Softmax over numbers that large is nearly one-hot. One token gets almost all the weight, and the gradient that reaches the other scores is close to zero, which slows learning. Dividing by $\sqrt{d_k}$ brings the variance back to about 1. In the demo $d_k = 2$ , so the effect is gentle; at $d_k = 64$ it is the difference between a model that trains and one that struggles.

What comes out

Attention’s output for each token is a new vector that mixes in information from the tokens it attended to. In a transformer this output is added back onto the token’s own vector, and the result moves on to the next part of the block. Stack enough layers and the vector for “bank” ends up different in “river bank” and “bank account”, even though both started from the same embedding.

Attention has no sense of orderOptional

Nothing in $\mathrm{softmax}(QK^\top/\sqrt{d_k})V$ refers to positions. If you shuffle the input tokens, the output rows shuffle in exactly the same way and are otherwise unchanged. Mathematically, self-attention is permutation equivariant. “Dog bites man” and “man bites dog” would produce the same set of vectors.

The same matrices $W_Q$ , $W_K$ , and $W_V$ are used at every position, and they are learned by gradient descent along with everything else in the network. Nothing about word order is built in, so it has to be added to the input. That is the subject of the lesson on position.

Key ideas

Each token gets a query, a key, and a value by multiplying its embedding by three learned matrices.
A token’s attention to another token comes from the dot product of its query with the other token’s key, scaled by $1/\sqrt{d_k}$ and normalized with softmax.
The output is a weighted average of the value vectors, so each token’s new vector carries information from the tokens it attended to.
In matrix form, $\mathrm{softmax}(QK^\top/\sqrt{d_k})V$ computes this for all tokens at once, at a cost that grows with the square of the sequence length.
Scaling by $\sqrt{d_k}$ keeps the softmax from saturating as the vectors get longer.