Your next step
Attention: match, weight and mix
Query, key, value, scores, shares, and the weighted mixture.
Start learningWhat you’ll learn
Start from one position
Identify which position is being updated in a toy attention example.
About 4 minutes · Open activity
A query describes what to match
Identify the query's role in an attention operation.
About 4 minutes · Open activity
Keys provide matching information
Identify what the query is compared against.
About 4 minutes · Open activity
Values carry the information to combine
Distinguish a key's role from a value's role.
About 4 minutes · Open activity
These roles come from learned transformations
Recognize that query, key and value strips are computed from representations.
About 4 minutes · Open activity
Compare the query with the keys
Read supplied compatibility scores.
About 4 minutes · Open activity
Keep score scale manageable
Explain the purpose of scaled scores without deriving a formula.
About 4 minutes · Open activity
Softmax turns scores into contribution shares
Identify softmax's role using the shares already learned.
About 4 minutes · Open activity
Combine the values
Trace the weighted mix to a new representation.
About 4 minutes · Open activity
Follow a new attention example
Identify query, keys and values, read supplied shares, and select the matching combined representation.
About 5 minutes · Open activity