Chapter C31 · Interactive Course
Language models have different designs
Encoder, decoder, encoder-decoder, embeddings, and the alternatives.
Core Learning Concept
Match the required result to the capabilities described in the fictional model cards. Separate representing an input, continuing a sequence, and transforming a source into an output. Architecture helps explain the route; actual task performance, interfaces, and limits still determine whether a candidate is useful.
What You’ll Learn in This Chapter
-
Encoder-only models represent an input
Recognize an encoder-style model used to classify or represent supplied text.
An encoder-style language model produces representations of an input, often using context on both sides of a position. A task-specific output component can turn those representations into a category or other result. Reading an input this way is different from repeatedly extending a sequence with new tokens.
-
Masked practice can use both sides
Identify which context supports a masked-text objective.
Masked-language training hides selected input tokens and asks the model to recover them. Visible context may lie on both sides of the hidden position. That differs from next-token prediction, which must not use the future target. The allowed information depends on the training objective, not a universal rule for all models.
-
Decoder-only models extend a sequence
Recognize a GPT-style causal generation route.
A decoder-only autoregressive model extends a sequence using causal attention. The prompt supplies initial context, then generated tokens are appended and used for later predictions. It does not require a separate source encoder. A surrounding application can still add retrieval, tools, or media-processing components.
-
Encoder-decoder models transform one sequence into another
Recognize a T5-style source-to-output design.
An encoder-decoder model represents a source sequence and generates a separate output sequence conditioned on it. Translation is a useful example: the full source can be encoded before the translation is produced. The separation describes the architecture; it does not alone establish translation quality.
-
Embedding models return representations
Distinguish a retrieval representation from generated prose.
An embedding model returns vectors intended for operations such as comparison or retrieval. Those numbers are not a prose answer. Query and document vectors must belong to compatible representation spaces, often produced by the same model or explicitly trained paired encoders. Arbitrary embeddings from unrelated models cannot simply be mixed.
-
Broad pretraining can support many tasks
Explain a foundation model as a reusable starting point.
A foundation model is broadly trained to serve as a versatile base for downstream tasks. Broad pre-training provides general capabilities, which can be adapted to specific applications using specialized prompting or targeted fine-tuning.
-
Transformers are not the only sequence design
Recognize recurrent, state-space and hybrid alternatives at a high level.
Sequence modeling includes recurrent, state-space, attention-based, and hybrid designs. They organize information and computation differently, with tradeoffs depending on the task and implementation. An architectural label is a starting point for understanding costs and behavior, not proof that one family always outperforms or undercuts every other family.
-
Read three fictional model cards
Match a representation task, a continuation task and a source-transformation task to compatible designs, and explain why architecture alone does not establish quality.