Post-Training | AI Engineering

What Post-Training Adds

Supervised fine-tuning teaches format and tasks from demonstrations. Post-training aligns behavior with human preferences: helpfulness, harmlessness, honesty, tone, and refusal policies. A model can ace SFT datasets yet still be sycophantic, overconfident, or unsafe without alignment stages.

Post-training is where products encode values: what the model should refuse, how it handles ambiguity, and whether it prioritizes brevity or thoroughness.

RLHF Pipeline

Reinforcement Learning from Human Feedback (RLHF) is the classic three-stage pipeline:

SFT on high-quality demonstrations (often already done)
Reward model (RM) training on human preference pairs (response A vs B for the same prompt)
RL fine-tuning (often PPO) to maximize reward while staying close to the SFT policy via a KL penalty

The reward model scores completions. The policy model generates text and receives reward signal. KL divergence to the reference model prevents collapse into high-reward gibberish or excessive length hacking.

Refer: InstructGPT / RLHF paper

Operational cost: RLHF is engineering-heavy: reward hacking, reward model drift, unstable PPO runs, and expensive human labeling. Many teams adopt simpler preference methods for iteration speed.

DPO and Preference Optimization Variants

Direct Preference Optimization (DPO) trains directly on preference pairs without an explicit RL loop. It reparameterizes the reward so the optimal policy satisfies a closed-form loss on chosen vs rejected responses. Benefits include simpler training, no separate reward model inference during alignment, and more stable convergence in many setups.

Variants (IPO, KTO, ORPO, SimPO) address length bias, unpaired data, or joint SFT-plus-preference objectives. The landscape evolves quickly, but the pattern remains: optimize likelihood of preferred outputs relative to dispreferred ones under a regularization constraint.

from trl import DPOTrainer

trainer = DPOTrainer(
    model=policy_model,
    ref_model=ref_model,
    train_dataset=preference_pairs,  # prompt, chosen, rejected
    tokenizer=tokenizer,
    beta=0.1,  # KL regularization strength
)
trainer.train()

Data: Preferences and Red Teaming

Preference data quality defines the ceiling. Labelers need clear rubrics: factuality, safety, style, and task completion. Diverse prompt distribution prevents the model from optimizing narrow benchmarks.

Red teaming proactively finds jailbreaks, bias, and capability gaps before launch. Findings feed new preference pairs, system prompt hardening, and safety classifiers. Alignment is iterative, not a one-shot training job.

Evaluating Alignment

Automatic metrics are incomplete. Teams combine:

Human side-by-side evals on production-like prompts
Safety benchmarks (toxicity, jailbreak success rate)
Capability retention checks (math, coding) to detect alignment tax
LLM-as-judge with human calibration (judge models drift and favor verbosity)

Alignment tax refers to capability loss after safety tuning. Mitigate with mixed SFT data, conservative KL penalties, and eval gates before shipping new checkpoints.

System Prompts vs Weight-Level Alignment

System prompts and guardrails shape behavior at inference without retraining. Weight-level alignment generalizes better to adversarial inputs and reduces reliance on prompt secrecy. Production systems use both: aligned weights as the foundation, runtime policies for product-specific rules, tool constraints, and dynamic retrieval context.