Chapter C96 · Interactive Course
Use it: fix a wrong answer three ways
Classify the failure, repair in the right layer, attribute the fix.
Core Learning Concept
Diagnose the supplied failure before choosing a repair. Check missing or stale information, candidate selection, and whether the model fits the task. Try a focused change and compare the same cases. Record what improved and the remaining limits so the repair can be evaluated and repeated.
What You’ll Learn in This Chapter
-
Name the failure first
Classify a wrong answer by its likely cause.
Describe the observable failure before naming its cause. An absent source, a stale source, an unsupported generated claim, and a capability mismatch require different tests. Trace what reached the model and how the output was produced. A diagnosis is a hypothesis to test, not something a bad answer proves by itself.
-
Try context before anything else
Fix a failure by supplying the missing material.
If needed information was missing, supplying a relevant excerpt is a focused first test. Keep the task and evaluation steady and compare the response. Improvement supports the usefulness of that repair in the tested setting; it does not prove that missing context was the only possible cause.
-
Retrieval for facts that change
Choose retrieval when currency is the problem.
Retrieval can supply current facts from maintained sources without retraining whenever a document changes. Check freshness, permissions, and whether the correct passage reaches the model. Retrieval is not sufficient on its own: the final answer still needs to preserve and cite what the source actually establishes.
-
Sampling for variation, not truth
Reject a temperature change as a fix for a wrong fact.
Sampling settings alter how candidate continuations are selected. They may change whether a particular mistake appears, but do not provide missing evidence or guarantee truth. Use them for the behavior they control, such as variation, and use sources or reliable tools to check factual correctness.
-
Smaller can be the right repair
Choose a simpler model when the task allows it.
A smaller or simpler model can be a useful repair when the task is narrow and constraints dominate. Test that it meets the required quality on representative cases. Lower cost is valuable after adequacy is established; choosing smaller without checking the task would repeat the same evidence gap.
-
Record what actually worked
Attribute a fix to the change that caused it.
A repair record states the observed problem, changed component, test cases, results, and limits. Keep comparisons interpretable by changing one factor where practical and accounting for output variability. Record unsuccessful trials too; they help distinguish a repeatable improvement from a lucky result.
-
Apply it: fix a wrong answer three ways
Carry out the whole decision once: classify a failure, try the cheapest repair, choose the right layer and attribute the fix.
-
Diagnose the whole answer pipeline
Find a retrieval failure, explain two sampling outcomes and compare a compressed model against a task requirement.