Interview: Post-Training, RLHF, DPO and Alignment

• 1. SFT - fine-tune the base model on demonstration data.
• 2. Reward model training - collect human preference comparisons (response A vs B), train a model to predict which one humans prefer.
• 3. RL fine-tuning - use the reward model as a signal to fine-tune the SFT model with an RL algorithm (typically PPO), while a KL-divergence penalty keeps the policy close to the SFT model so it doesn't drift into degenerate high-reward-but-nonsensical outputs.

A reward model takes a prompt and response and outputs a scalar score estimating human preference. It's trained on pairs of responses ranked by human annotators using a Bradley-Terry-style loss that pushes the score of the preferred response higher than the rejected one.

DPO reformulates the RLHF objective mathematically so that the optimal policy can be derived directly from preference data using a simple classification-style loss, without needing to train a separate reward model or run unstable RL optimization loops. It's simpler to implement, more stable to train, and requires less compute/infrastructure, though PPO-based RLHF can still edge out DPO in some settings with careful tuning.

Without a constraint, the policy can learn to "reward hack" - find outputs that score highly on the (imperfect) reward model but are actually low quality, repetitive, or nonsensical. Penalizing KL-divergence from the original SFT model keeps updates close to a known-good distribution, trading some reward-maximization for reliability.

RLAIF (Reinforcement Learning from AI Feedback) uses another (often larger/more capable) AI model to generate preference labels instead of, or alongside, human annotators, which is cheaper and more scalable but depends on the labeling model's own biases and blind spots rather than direct human judgment.

An alignment approach where a model critiques and revises its own outputs against a set of written principles ("constitution"), generating a self-improved dataset used for further training, reducing reliance on large volumes of human-labeled harmful-content examples.

ORPO combines SFT and preference optimization in one stage without a reference model. KTO (Kahneman-Tversky Optimization) learns from binary good/bad labels without paired comparisons. GRPO (Group Relative Policy Optimization) is used in recent RL-style post-training (e.g. DeepSeek-R1) with group-relative rewards instead of a separate critic. All aim to simplify or stabilize alignment vs classic PPO+reward model.

Reward hacking is when the policy exploits flaws in the reward model (verbosity, format tricks, toxic fluency) to score high without being actually helpful. Detect via human eval on RL outputs, KL drift monitoring, adversarial prompts, and comparing win-rate vs a held-out human-labeled set the reward model never saw.