Chapter C30 · Interactive Course
Build the Transformer picture
The block, the stack, and what 2017 actually contributed.
Core Learning Concept
Trace the original Transformer from source tokens to output. Locate attention beside feed-forward layers, residual shortcuts, and normalization. Distinguish parallel training over known sequences from generating new output tokens one by one.
What You’ll Learn in This Chapter
-
Attention is one part of a block
Distinguish information mixing from the whole model.
A Transformer block combines attention with feed-forward layers, residual links, and normalization. Attention mixes tokens across positions, while surrounding layers refine each representation independently, creating a complete and balanced processing unit.
-
A feed-forward step transforms each position
Identify a position-wise network after attention.
A position-wise feed-forward network processes each token position independently using shared layer parameters across the sequence. While attention shares information between positions, this feed-forward step refines each token internal features separately.
-
A residual route preserves a path
Recognize a skip connection that combines earlier and transformed information.
A residual connection adds a layer input directly to its transformed output, forming x + f(x). This bypass shortcut preserves original signal flow and eases gradient propagation across deeply stacked network layers.
-
Normalization manages internal scale
Explain normalization's supporting role without deriving it.
Layer normalization rescales internal numerical activations across features to keep activations balanced. Stabilizing value ranges prevents exploding or vanishing numbers, allowing deep Transformer networks to train reliably across many layers.
-
Stack blocks to refine representations
Trace several blocks without treating each as a different agent.
Stacked Transformer blocks repeatedly transform representations across successive layers. Each block holds its own learned parameters, gradually building richer abstract features from low-level token embeddings up to high-level semantic representations.
-
The encoder represents the source
Identify the source side of the original translation Transformer.
In an encoder-decoder Transformer, the encoder processes the entire source sentence using bidirectional self-attention. The resulting contextual representations are then passed forward to the decoder to guide and condition output generation.
-
The decoder produces the next piece
Trace masked output context and source access in the original design.
The Transformer decoder pairs causal self-attention over previously generated tokens with cross-attention to encoder representations. A final output projection then predicts probabilities for the next token in the vocabulary at each step.
-
Training many positions differs from generating a reply
Distinguish parallel position computation from autoregressive generation.
Training calculates all sequence positions simultaneously in parallel using causal masking. In contrast, actual generation must produce tokens sequentially one after another, as each new token depends on previously generated outputs.
-
What Attention Is All You Need contributed
Identify the paper's historical claim in its original context.
The 2017 paper introduced an all-attention sequence architecture that eliminated recurrent steps. While earlier models used attention as a secondary aid, Transformers established attention as the central mechanism powering modern architectures.
-
Explain the Transformer to a friend
Trace source to output on a supplied diagram, explain masking, and distinguish the 2017 contribution from earlier attention.