Your next step

From text to small pieces

Tokens, identifiers, and what a budget is counted in.

What you’ll learn

A token is a piece, not always a word
Identify pieces in a supplied tokenization.
About 4 minutes · Open activity
The tokenizer makes the pieces
Explain why two tokenizers can split the same text differently.
About 4 minutes · Open activity
A token ID is an address
Distinguish a lookup identifier from meaning.
About 4 minutes · Open activity
Small pieces handle unfamiliar words
Explain the purpose of subword representation.
About 4 minutes · Open activity
Language changes the space needed
Recognize that equal-looking text can use different token budgets.
About 4 minutes · Open activity
A context window has a budget
Identify which supplied material must fit a stated input/output limit.
About 4 minutes · Open activity
Pack a tiny context tray
Use supplied token counts to fit a short request and a relevant note, and identify why a token ID is not its meaning.
About 5 minutes · Open activity