Chapter C40 · Interactive Course
What does better actually mean?
Criteria, precision, recall, thresholds, subgroups and sample size.
Core Learning Concept
Read scorecards from their underlying counts. Calculate precision and recall with proper denominators, inspect key subgroups, and assess sample sizes. A single headline accuracy score never tells the whole story.
What You’ll Learn in This Chapter
-
Start with a useful success criterion
Choose a metric that matches the actual task.
A success criterion states what the system must achieve for the task. For an urgency sorter, missing an urgent message may matter more than putting an ordinary message in the review queue. Define those consequences before selecting metrics. Style can also be a criterion when communication quality is part of the job.
-
Accuracy can hide rare failures
Recognize an unhelpful high accuracy score on imbalanced examples.
Accuracy is the fraction of all cases classified correctly. If only one of one hundred messages is urgent, predicting “not urgent” for everything gives 99% accuracy while missing every urgent message. Inspect class-specific counts so a large easy group does not hide the failure you care about.
-
Precision asks about the flagged cases
Interpret precision from a supplied visual count.
Precision asks: of all cases flagged positive, how many were truly positive? If 10 items were flagged and 8 were correct, precision is 80%. It evaluates the flagged set and ignores missed positives outside it.
-
Recall asks about all relevant cases
Interpret recall from a supplied visual count.
Recall asks: of all the truly positive cases, how many were found? If twelve messages were urgent and eight were flagged, recall is 8/12, about 67%. Use the true-positive population as the denominator. Read precision alongside recall to understand both false alarms and missed cases.
-
Changing a threshold changes the errors
Explain a false-alarm versus missed-case tradeoff.
A threshold converts a score into a decision, such as flagging messages above 0.7. For a fixed scoring model, lowering the threshold generally finds more positives while adding false alarms. Choose it using the costs and benefits of errors, and evaluate the resulting counts on representative data.
-
Test the people and situations that matter
Identify a group or condition hidden by an average.
Always evaluate the actual inputs and situations your system will encounter. A strong overall score often masks serious weaknesses in specific conditions or user groups. Report sample sizes and context for every comparison.
-
A benchmark is a sample of tasks
Identify what a published score leaves outside its scope.
A benchmark is a defined collection of tasks and scoring conditions used for evaluation. Its score provides evidence about that sample and setup. Inspect what it includes, what it omits, and whether the system had access to similar examples. Transfer to your workflow remains something to test.
-
Small differences need enough evidence
Recognize uncertainty in a tiny comparison.
Observed differences contain uncertainty from the sample and sometimes from generation or judging variability. One extra success in ten trials may not support a broad ranking. Report counts, choose enough relevant cases for the decision, and use appropriate uncertainty analysis rather than a universal minimum sample-size rule.
-
Read a scorecard carefully
Inspect a fictional classifier's flags, missed cases and subgroup results, and explain why the headline score is insufficient.