Chapter C29 · Interactive Course
Attention across a sequence
Self and cross attention, order signals, masking, many heads.
Core Learning Concept
Compare attention diagrams by the origin of queries, keys, and values. Distinguish self-attention from cross-attention and inspect the causal mask. Remember that attention heatmaps show internal routing rather than complete explanations of model decisions.
What You’ll Learn in This Chapter
-
Self-attention uses one sequence
Identify where queries, keys and values originate.
Self-attention forms queries, keys, and values from representations of the same sequence. Each allowed position can contribute information to another position update, allowing the model to build contextual representations across the whole input sequence.
-
Each position gets its own update
Recognize that attention is repeated across positions.
Every query position computes its own unique distribution of attention shares across all keys. In a heatmap, rows represent query positions and columns represent candidate keys. The normalized weights within each row always sum to one.
-
Representations become contextual
Explain a token's changed representation after incorporating other positions.
A contextual representation incorporates information from the surrounding sequence into a token vector. The word bank updates differently near river than near deposit. This dynamic numerical update reflects context without changing stored vocabulary definitions.
-
Positions need an order signal
Explain why a Transformer needs position information.
Standard dot-product attention ignores word order because sets are unordered. Transformers inject positional encodings into token vectors so the network can distinguish identical words appearing in different arrangements across the sequence.
-
A causal mask prevents looking ahead
Identify legal context for next-token prediction.
Autoregressive generation produces tokens one at a time. A causal attention mask prevents earlier positions from viewing future tokens during training, mirroring real-world inference where future tokens do not yet exist.
-
Cross-attention connects two sequences
Identify attention from an output position to source representations.
Cross-attention takes queries from the target sequence and keys and values from the source sequence. In encoder-decoder translation, output positions use cross-attention to inspect relevant source tokens directly throughout generation.
-
Multiple heads offer several learned views
Explain why heads can combine different relationships.
Multi-head attention applies several independent projection sets to queries, keys, and values in parallel. Multiple heads capture distinct contextual relationships simultaneously, combining their individual outputs into one richer composite vector representation.
-
A heatmap is only one view
Interpret an attention picture with appropriate limits.
An attention heatmap displays computed weights for a specific head and layer. While heatmaps illustrate internal weighting patterns, they do not provide a complete causal explanation for why the model produced a particular final answer.
-
Compare three attention diagrams
Classify self versus cross attention, block an illegal future link, and state what a heatmap cannot establish.