Chapter C25 · Interactive Course
From text to small pieces
Tokens, identifiers, and what a budget is counted in.
Core Learning Concept
Fit tasks within documented token budgets using precise piece counts. Reserve required output room, preserve key source material, and trim low-priority content rather than relying on rough word counts or estimates.
What You’ll Learn in This Chapter
-
A token is a piece, not always a word
Identify pieces in a supplied tokenization.
A token is a unit of text used by a model’s tokenizer. It can be a word, part of a word, punctuation, or a space-related piece. The split shown is an illustration for the specified tokenizer. Count pieces rather than assuming one visible word always occupies one token.
-
The tokenizer makes the pieces
Explain why two tokenizers can split the same text differently.
A tokenizer converts raw text into model-specific pieces. Because tokenizers use different vocabulary rules, the same sentence produces varying token counts across systems. Always check limits using the intended model's specific tokenizer.
-
A token ID is an address
Distinguish a lookup identifier from meaning.
A token ID is an identifier used to look up the representation of a token. Its numeric size does not express importance or meaning. ID 900 is not necessarily more similar to 901 than to 12. Compare learned representations for relationships, not the arbitrary numbering of vocabulary entries.
-
Small pieces handle unfamiliar words
Explain the purpose of subword representation.
Subword tokenization represents many unfamiliar words using familiar smaller pieces. A new name may be split into several units instead of requiring its own vocabulary entry. Exact coverage depends on the tokenizer, including any byte fallback. Representing the spelling does not establish knowledge about the person or concept named.
-
Language changes the space needed
Recognize that equal-looking text can use different token budgets.
Token counts vary across languages and tokenizers. Equivalent messages require different numbers of pieces, affecting context capacity and usage costs. Measure representative examples directly instead of assuming a single universal conversion ratio.
-
A context window has a budget
Identify which supplied material must fit a stated input/output limit.
A context budget limits available model space across instructions, conversation, source text, and generated replies. Services publish these rules in an API. Monitor total token consumption so requests do not exceed documented boundaries.
-
Pack a tiny context tray
Use supplied token counts to fit a short request and a relevant note, and identify why a token ID is not its meaning.