Chapter C27 · Interactive Course
The problem attention helps solve
Order, distance, and the fixed-summary bottleneck.
Core Learning Concept
Find the clue needed to interpret the supplied sequence, even when it is far from the ambiguous word. Compare retaining information in a running state with revisiting source-position representations. Explain what attention contributes, while keeping the computational mechanism separate from human awareness.
What You’ll Learn in This Chapter
-
Order changes the message
Identify a relationship changed by rearranging words.
A sequence is an ordered collection. Changing the order of words can change who did what, even if the same words remain. A model therefore needs a way to represent order as well as token identity. The examples compare sentences whose difference lies in arrangement rather than vocabulary.
-
A useful clue can be far away
Identify a relevant distant part of a sequence.
Relevant context often sits far from an ambiguous word in a passage. Resolving pronouns requires finding the true supporting clue rather than assuming the nearest noun supplies the correct reference.
-
Reading with a running state
Explain the basic idea of a recurrent network.
A recurrent network processes a sequence through repeated updates to a state. Each new input and the previous state produce the next state. The state can carry information forward, but it has limited capacity and may fail to preserve needed details. It is not a lossless record of everything read.
-
Some recurrent designs manage what to retain
Recognize the purpose of gates in an LSTM-style model.
Gated recurrent architectures like LSTMs regulate information flow with learned gates. These controls decide what to retain, update, or discard, preserving long-range signals across sequential steps through purely numerical operations.
-
Compressing a sentence can lose useful detail
Explain the motivation for revisiting source positions.
Early sequence models squeezed entire source texts into a single fixed vector bottleneck. Attention overcomes this constraint by allowing output steps to inspect and combine representations from multiple source positions directly.
-
Earlier attention connected source and output
Place the 2014 attention work before the 2017 Transformer paper.
The 2014 Bahdanau model introduced attention into recurrent translation to align source words dynamically. In 2017, the Transformer replaced recurrent layers entirely with attention mechanisms, establishing today's modern architectural foundations.
-
Attention is computation, not awareness
State the boundary of the attention metaphor.
Attention is a computation that scores relationships and mixes representations using weights. The name suggests selective focus, but it does not establish awareness or conscious noticing. A visualization shows aspects of those numerical operations. It should not be treated as a complete explanation of why the model produced an answer.
-
Solve the missing-context puzzle
Find a distant clue, explain the summary bottleneck, and place earlier attention before the Transformer milestone.