Chapter C45 · Interactive Course
Why responses take time and cost money
Prefill, decode, caches, memory limits and end-to-end budgets.
Core Learning Concept
Read the receipt by separating input processing, output generation, cache use, and repeated attempts. Then calculate the cost of the completed task rather than one isolated call. Use the supplied measurements to identify a practical saving and state which other costs or delays the receipt does not expose.
What You’ll Learn in This Chapter
-
Prefill reads the input
Identify processing that happens before the first generated token.
Prefill is the processing of supplied input before autoregressive output generation. Longer inputs can require more work, though hardware, batching, and caching affect the time. Time to first output can also include network and queue delays. Use the trace to separate these contributors rather than attributing the whole wait to reading.
-
Decode generates the continuation
Distinguish generation steps from input processing.
Decode is the repeated generation of continuation tokens in an autoregressive model. Each new token becomes context for later steps. Streaming can expose output while this process continues, but products may buffer chunks. A visible pause may come from generation, tools, or delivery, so inspect the relevant measurements.
-
A KV cache reuses attention work
Explain why stored keys and values can avoid repeated computation.
A key-value cache stores attention vectors from prior token positions. In standard causal decoding, appending a token reuses past keys and values without recomputing them. This runtime cache uses GPU memory to accelerate generation.
-
Prompt caching and answer caching differ
Distinguish reused prefix computation from reusing a previous response.
Prompt or prefix caching reuses compatible computation for repeated input while still allowing a newly generated response. Answer caching returns a previously stored result. The two have different invalidation and freshness requirements. Inspect which cache is involved before expecting a cost saving or trusting an old answer.
-
Memory capacity can limit a run
Identify weights and temporary state as separate memory demands.
Runtime memory can include model parameters, activations, key-value caches, and implementation overhead. Longer context and more concurrent requests can increase requirements. A model’s supported context limit does not mean every device can run that context comfortably. Evaluate memory under the actual model, precision, hardware, and workload.
-
A local interface can call a remote model
Trace where inference and tools actually run.
A local interface is a program running on your device. Its model inference may still occur remotely, and connected tools may contact additional services. These are separate facts. Trace the configured provider and connections before concluding that installing an app locally keeps all processing and data on the device.
-
Budget the whole task
Account for input, output, tool work and retries.
True task cost includes failed runs, retries, input context, outputs, and external tool calls. A cheaper token price often ends up costing more overall if success rates are poor. Always calculate total cost per successful outcome.
-
Read a request receipt
Locate prefill, decode, cache use and repeated work in two fictional receipts, and explain the tradeoff without naming a winning model.