Your next step
From hidden states to an answer
Vocabulary, scores, the output head, targets and stop reasons.
Start learningWhat you’ll learn
The output vocabulary
Identify the set of token choices at a generation step.
About 4 minutes · Open activity
Logits are scores before probabilities
Distinguish raw output scores from normalized probabilities.
About 4 minutes · Open activity
The language-model head
Locate the mapping from a hidden representation to vocabulary scores.
About 4 minutes · Open activity
Predicting the next training token
Identify the target used at one position during next-token training.
About 4 minutes · Open activity
Token loss rewards the observed target
Explain why assigning the observed target less probability increases loss.
About 4 minutes · Open activity
A turn can stop for different reasons
Distinguish a natural stop from an output-budget limit.
About 4 minutes · Open activity
Remembered patterns can include memorization
Distinguish learned generalization from reproducing training material.
About 4 minutes · Open activity
Trace one generated token
Label representation, output scores, sampling choice and completion status in a fictional decoder trace, and identify one uncertainty about provenance.
About 5 minutes · Open activity