Chapter C97 · Interactive Course
Use it: run a fair comparison
One task, one criterion, conditions held, and an honest limit.
Core Learning Concept
Run a fair small comparison with a stated task and rubric. Keep relevant conditions comparable, include representative difficult cases, and report raw counts and limitations. If evidence does not distinguish the options, you can choose on a supported practical factor while recording what further testing would be needed.
What You’ll Learn in This Chapter
-
One task, one criterion
State the single question a comparison answers.
Define the task and primary success criterion before collecting outputs. “Summarize this notice while preserving every exception” is testable; “which model is better?” leaves the evaluation undefined. Add secondary constraints such as cost or latency when relevant, and avoid hiding them inside an unexplained overall score.
-
Hold the conditions still
Keep everything but the compared thing constant.
Hold all background conditions constant when comparing models: prompts, tools, and evaluation criteria. If setups differ arbitrarily, report a whole-system comparison rather than attributing differences to raw model capability alone.
-
Enough cases to support the decision
Report uncertainty and choose more evaluation cases when a result is inconclusive.
Required sample size depends on task variability and decision stakes. Five test cases can catch blatant errors but cannot prove subtle performance differences. Always report sample counts and selection methods rather than relying on gut feel.
-
Write the rubric first
Define the judgment before seeing outputs.
A rubric states how an output will be judged before you see the results. Define acceptable behavior and important failures with examples where useful. Apply the same rubric to every candidate. Clear criteria reduce the chance of adjusting the evaluation to favor an answer you already like.
-
Report what you cannot conclude
State the limit of your own comparison.
Report the boundaries of the comparison: tasks, languages, conditions, sample size, and unmeasured requirements. Explain how those limits affect the decision and name the next useful test. A narrow, accurate result can guide action more reliably than a broad claim unsupported by the trial.
-
Apply it: run a fair comparison
Carry out the whole decision once: state the question, hold conditions still, use enough cases, judge by a rubric written first, and report the limits.
-
Review a model release with evidence
Inspect a fictional model card, scorecard and release report; make a conditional use decision and specify follow-up evidence.