Chapter C38 · Interactive Course
What happens while an answer is generated
Distributions, greedy, sampling, temperature, top-p, caching.
Core Learning Concept
Compare the supplied generation settings and follow how they change candidate selection. Distinguish model scores, sampling choices, prompt examples, and reused computation. Explain differing replies without assuming that ordinary inference retrains the model. For factual tasks, keep verification separate from variation controls.
What You’ll Learn in This Chapter
-
The next piece has a distribution
Read a supplied set of candidate-token probabilities.
At a generation step, a language model produces scores that can be converted into a distribution over possible next tokens. The distribution describes predicted continuations under the setup. It is not a probability that an entire answer is true. A likely next word can continue an incorrect sentence.
-
Greedy choice takes the top candidate
Recognize greedy decoding.
Greedy decoding selects the highest-scoring next token at each step. Given identical scores and tie handling, the choice is fixed, but a deployed service may still vary because of other factors. Locally preferred tokens do not guarantee the best whole response or a factually correct answer.
-
Sampling allows other likely candidates
Explain why the same starting context can yield variation.
Sampling draws a next token according to a chosen distribution instead of always taking the top one. Repeating the same input can therefore produce different continuations. This can be useful for exploring alternatives. Variation alone does not indicate that the model learned something between the requests.
-
Temperature reshapes the distribution
Interpret a temperature change in a fixed illustrative sampling setup.
Temperature divides and scales raw logit scores before softmax normalization. Lower temperatures concentrate probability sharply on top candidates, yielding predictable outputs, while higher values flatten probabilities to encourage diverse continuations.
-
Top-p limits the candidate pool
Explain nucleus sampling using a supplied cumulative share.
Top-p, or nucleus sampling, restricts sampling to a leading set whose cumulative probability reaches a threshold. If the leading token already has most of the mass, the set can be small. The threshold changes the candidate pool; it does not measure confidence that the resulting claim is true.
-
Examples in a prompt can guide a response
Distinguish in-context adaptation from fine-tuning.
Examples in a prompt demonstrate the desired input-output pattern within the current context. This is often called few-shot prompting when several examples are supplied. It influences inference without ordinarily updating model parameters. Check example relevance and correctness, and account for their context cost.
-
Reuse earlier computation
Explain the purpose of an inference cache at a high level.
Autoregressive models can cache intermediate information from earlier positions to avoid recomputing it for each new token. A key-value cache is one such mechanism. It uses memory to save computation under valid reuse conditions. This technical cache is distinct from a product remembering personal preferences across conversations.
-
More attempts need a useful selection rule
Explain the limit of inference-time candidate search.
Multiple attempts help only when the overall method improves the selected result enough to justify the cost. A verifier, rubric, or aggregation rule can guide selection; each has limits. Majority agreement can be useful on some tasks but can also reinforce a shared mistake. Evaluate the full procedure.
-
Explain two different replies
Use a fixed-model scenario to identify sampling, prompt examples and extra checking as different causes of output change.