Chapter C24 · Interactive Course
Learning that works on new examples
Judge generalisation with separate evidence.
Core Learning Concept
Compare training and validation results before choosing a checkpoint, then inspect whether the final test is independent and relevant. Look for underfitting, overfitting, leakage, and changed conditions as different possible problems. Choose a repair supported by the evidence rather than applying the same remedy to every low score.
What You’ll Learn in This Chapter
-
The words a training run uses
Read a training chart by naming its parameters, loss, epochs and learning rate, and separate a chosen setting from an adjusted number.
Parameters are the numbers training adjusts. Loss is one chosen measure of how wrong the predictions were. An epoch is one pass through the training examples. The learning rate is a setting fixed beforehand. The two optional chapters before this one show how each is computed; this chapter only needs you to read them off a chart.
-
Training examples are not the final exam
Identify why evaluation needs unseen examples.
Generalization means performing accurately on fresh, unseen examples. Training metrics alone cannot demonstrate this because models can memorize training instances. Always evaluate on truly held-out data representing the real deployment task.
-
Validation helps choose; testing checks later
Assign training, validation and test sets distinct roles.
Training data updates parameters. Validation data helps select settings or checkpoints. Test data provides a later evaluation of the selected system. Repeated decisions based on a test set make it part of development. Keep those roles explicit so the final result does not quietly report the best of many attempts.
-
Too little learned structure
Recognize underfitting from supplied error patterns.
Underfitting means the model has not captured enough useful structure for the task. Poor results on both training and validation can suggest it, but also check noisy labels, weak features, and training problems. A suitable repair depends on the cause; simply making every model larger is not a diagnosis.
-
Remembering quirks instead of patterns
Recognize overfitting from divergent training and validation results.
Overfitting occurs when learning fits details of the training sample that do not transfer well. Falling training loss alongside rising validation loss is a common warning sign. Compare the curves rather than choosing by training loss alone. A gap suggests investigation; its cause and practical importance still need examination.
-
Discourage brittle fits
Explain regularization as a way to constrain learning.
Regularization constrains training to discourage fragile, overly complex fits. Techniques like weight penalties and dropout prevent over-reliance on individual parameters. Tune regularization strength on validation data to balance model capacity against stability.
-
Know when to stop training
Select a checkpoint using supplied validation evidence.
Early stopping continuously monitors validation loss to halt training when performance ceases improving on held-out data. Preserving the best intermediate checkpoint prevents the overfitting that occurs when extended training degrades new-case accuracy.
-
When the answer leaks into the test
Identify data leakage from a duplicated or unavailable clue.
Data leakage gives development or evaluation information that should not be available in the intended use. Examples include related records crossing a split or a feature recorded only after the outcome. A high score can be correctly calculated yet misleading because the evaluation setup made the task artificially easy.
-
The world can change after testing
Recognize distribution shift.
Distribution shift means the deployed conditions differ from those represented during development or evaluation. Daylight photographs may not establish performance at night. Changes can affect inputs, relationships, or usage. Monitor the actual setting and obtain relevant new evaluation evidence instead of assuming an old score applies indefinitely.
-
Choose the trustworthy training result
Inspect training and validation curves, catch an overlapping test example, and name a fresh evaluation for a new setting.