Chapter C35 · Interactive Course
Learning from feedback
Preferences, reward models, RLHF, DPO, and reward gaming.
Core Learning Concept
Inspect feedback as a training signal with a purpose and limitations. Read the rubric, compare what reviewers reward, and look for shortcuts that improve the score without improving the real result. Use independent checks where possible and keep both harmful responses and unnecessary refusals in view.
What You’ll Learn in This Chapter
-
People can compare two responses
Identify preference data rather than a single correct answer.
Preference data records that one response was preferred to another under a review process. It does not necessarily identify a universally correct answer. Inspect the criteria: a reviewer might prioritize correctness, clarity, or brevity. Those instructions help explain what the preference label actually teaches.
-
A reward model predicts preference
Distinguish a learned score from a verified truth judgment.
A reward model is a learned scorer trained to estimate a desired evaluation signal, often human preferences. Its output can guide further optimization. Because it is a model, it can make mistakes or reward shortcuts. A higher reward score needs validation against the underlying behavior people wanted.
-
RLHF uses human feedback to guide training
Trace preference evidence into a further-training signal.
Reinforcement learning from human feedback, or RLHF, uses human-derived feedback to guide training. A common route trains a reward model from comparisons and then optimizes a policy using its scores. Actual pipelines vary. A product thumbs-up may be collected for later use; it is not an immediate parameter update.
-
DPO learns directly from preference pairs
Recognize a different route for preference-based adaptation.
Direct Preference Optimization adjusts model parameters directly from preference pairs using an analytic loss function. It increases the likelihood of preferred completions without training an intermediate reward model or running complex reinforcement learning loops.
-
AI-generated feedback still has human choices behind it
Explain what shapes RLAIF or constitutional feedback.
AI-generated feedback can scale evaluation or training, but human choices remain in the criteria, model selection, examples, and acceptance rules. Automated judgments can share errors with the model being judged. Audit representative cases and compare against trustworthy references or human review where the task requires it.
-
Some outcomes can be checked automatically
Recognize verifiable rewards on a bounded task.
Some tasks have automatic checks, such as executing a test or comparing an arithmetic result. These can provide stronger signals for a defined property than preference alone. The check still has a scope and may be incomplete or exploitable. Passing it does not prove every aspect of the outcome is correct.
-
Optimizing the score can miss the intention
Identify reward gaming in a novel scenario.
Optimizing a score can exploit differences between the metric and the intended goal. If reviewers reward length, a model may become verbose without becoming more helpful. Identify the shortcut, revise the criteria, and test cases where the shortcut fails. Measure the desired outcome as directly as practical.
-
Safer behavior needs continuing evaluation
Distinguish trained refusal behavior from guaranteed safety.
Safer behavior requires repeated evaluation across relevant situations and changes. A system can fail by complying with a harmful request or by refusing a legitimate one. Inspect both, together with severity and coverage. One aggregate number cannot fully describe the safety and usefulness of a deployed assistant.
-
Improve the feedback rubric
Compare two response preferences, identify a reward shortcut, and propose a bounded check that does not confuse approval with truth.