Chapter C28 · Interactive Course
Attention: match, weight and mix
Query, key, value, scores, shares, and the weighted mixture.
Core Learning Concept
Follow one attention calculation using the values supplied. Choose the position being updated, compare its query with available keys, convert scores to shares, and mix values proportionally. Trace what each part contributes carefully.
What You’ll Learn in This Chapter
-
Start from one position
Identify which position is being updated in a toy attention example.
Identify the sequence position whose representation is being updated. It receives a weighted mixture of information from allowed positions. Following a single position calculation first makes the full attention diagram straightforward to trace.
-
A query describes what to match
Identify the query's role in an attention operation.
A query is a vector computed from the position being updated, indicating what features to seek. Comparing this query with candidate keys produces compatibility scores. This internal numerical representation differs entirely from an end-user’s prompt.
-
Keys provide matching information
Identify what the query is compared against.
A key is a vector representing a candidate position for matching. Comparing a query against candidate keys computes compatibility scores. Keys do not carry human labels; their projection transforms are learned during training.
-
Values carry the information to combine
Distinguish a key's role from a value's role.
The value is the content vector mixed into the result. While keys determine how much a position contributes, values provide what it actually contributes. Each value vector is multiplied by its calculated attention share.
-
These roles come from learned transformations
Recognize that query, key and value strips are computed from representations.
Queries, keys, and values are computed by applying learned projection weights to input representations. The same position supplies three separate vectors with distinct roles, allowing the model to decouple matching criteria from transferred content.
-
Compare the query with the keys
Read supplied compatibility scores.
A compatibility score measures alignment between a query and a key, commonly via dot product. Higher scores yield larger relative shares. This reflects learned numerical alignment rather than human truth or factual correctness.
-
Keep score scale manageable
Explain the purpose of scaled scores without deriving a formula.
Scaling divides dot products by the square root of key dimensions to keep score sizes manageable. Without scaling, extreme scores flatten softmax gradients. Scaling stabilizes internal training without altering the core matching objective.
-
Softmax turns scores into contribution shares
Identify softmax's role using the shares already learned.
Softmax converts raw compatibility scores into positive shares that sum to one. Greater scores yield larger mixing proportions. These dynamic weights are computed on the fly for each input rather than stored as parameters.
-
Combine the values
Trace the weighted mix to a new representation.
The attention output is a weighted sum of value vectors across all positions. Each value is multiplied by its attention share and summed, producing a newly updated contextual vector for the target position.
-
Follow a new attention example
Identify query, keys and values, read supplied shares, and select the matching combined representation.