Post-Training and Alignment

Starting From Basics: Why SFT Is Not Enough

A supervised fine-tuned (SFT) model has learned to imitate the completions in its training set. Imitation is a powerful starting point, but it has a structural ceiling: the model can only ever be as good as the demonstrations it was shown, and imitation gives the model no notion of "this answer is better than that answer," only "this is a valid answer to produce." Two consequences follow directly from this.

First, an SFT model has no mechanism for comparing outputs. If a prompt has two very different but both individually plausible completions in the training data (one terse and correct, one long-winded and slightly wrong), the model learns to imitate both patterns with roughly equal weight, and at inference time it may reproduce either. There is no gradient signal anywhere in SFT that says "prefer the terse, correct one."

Second, imitation of human-written text tends to inherit human writing habits that are not actually what you want from an assistant: hedging, excessive agreement with the user even when the user is wrong, verbosity used to appear thorough, and confident-sounding answers to questions the model does not actually know the answer to, because confident phrasing is what the training demonstrations looked like. None of these are bugs the SFT objective can see, because the SFT objective only measures similarity to a single reference completion, never quality relative to alternatives.

Post-training, sometimes called the alignment stage, is the set of techniques applied after SFT to correct exactly this gap. Instead of training on single demonstrations, post-training trains on comparisons: given a prompt and two (or more) candidate responses, which one is better, and by extension, adjust the model so it becomes more likely to produce the better kind of response in the future. This is where the properties users associate with "a good assistant" actually get instilled: helpfulness, honesty (including admitting uncertainty), harmlessness, appropriate tone, and a clear refusal policy for unsafe requests. A model can score extremely well on SFT training metrics and still be sycophantic, overconfident, or willing to produce harmful content, precisely because none of those failure modes are visible to a pure imitation objective. Post-training is also where a product encodes its specific values: what exactly counts as an unsafe request worth refusing, how the model should behave under ambiguous instructions, and whether the product favors brevity or thoroughness by default.

The Reward Modeling Problem

Before any policy can be optimized against preferences, you need a way to turn "humans prefer response A over response B" into a differentiable training signal. This is the job of the reward model (RM): a separate model, usually initialized from the same pretrained or SFT checkpoint as the policy, that takes a prompt and a response and outputs a single scalar score.

Human annotators are shown a prompt with two (or more) candidate completions, generated by sampling the current policy, and asked to indicate which one they prefer, according to a rubric covering factuality, helpfulness, safety, and style. This produces a dataset of triples (x, y_w, y_l), meaning prompt x, the winning (preferred) response y_w, and the losing (rejected) response y_l.

The reward model is trained under the Bradley-Terry model of pairwise comparison, which assumes the probability that a human prefers y_w over y_l is a logistic function of the difference in their underlying "true" scores:

P(y_w > y_l | x) = σ( r_φ(x, y_w) - r_φ(x, y_l) )

The reward model's parameters φ are trained to maximize the likelihood of the observed human preferences under this model, which is equivalent to minimizing the negative log-likelihood:

L(φ) = - E_{(x, y_w, y_l)} [ log σ( r_φ(x, y_w) - r_φ(x, y_l) ) ]

Notice this loss only depends on the difference between the two scores, never on their absolute magnitude. This is an important and sometimes underappreciated property: the reward model has no calibrated notion of "how good" a response is in any absolute sense, only a relative ranking. Two reward models trained on the same data can produce wildly different absolute score ranges while inducing the exact same ranking over responses, because the loss is invariant to any monotonic shift applied identically to both terms.

Reward model quality is bottlenecked by several practical issues that matter as much as the math: annotator disagreement (different people genuinely prefer different things, and averaging over annotators smooths out real, valid diversity of preference), annotator fatigue and inconsistency over long labeling sessions, and the prompt distribution used to generate comparison pairs, since a reward model trained mostly on comparisons for coding prompts will not necessarily generalize its judgments to creative writing prompts.

RLHF: The Classical Three-Stage Pipeline

Reinforcement Learning from Human Feedback (RLHF), as introduced in the InstructGPT line of work, chains together three distinct training stages:

1. SFT. Train (or start from) a model on high-quality human demonstrations, as covered in supervised fine-tuning. This produces the reference policy π_SFT and the initialization point for the next two stages.
2. Reward model training. As described above, train r_φ on human preference comparisons, typically sampled from completions generated by π_SFT itself so the reward model sees the actual output distribution it will later be scoring.
3. RL fine-tuning. Use the trained reward model as the optimization signal to further update the policy, most commonly with Proximal Policy Optimization (PPO), while constraining the policy to stay close to π_SFT.

The full RL objective being maximized is:

J(θ) = E_{x~D, y~π_θ(·|x)} [ r_φ(x, y) ] - β · E_x [ KL( π_θ(·|x) ‖ π_SFT(·|x) ) ]

The first term says: sample a prompt from the training distribution, sample a response from the current policy, and maximize the reward model's score for that response. The second term is a KL-divergence penalty against the original SFT policy, scaled by a coefficient β. This penalty is not optional decoration, it is structurally necessary for the whole procedure to work.

Without the KL term, the policy is free to move arbitrarily far from any distribution a human would recognize as reasonable text, as long as doing so increases the score assigned by the reward model. Because the reward model is a learned approximation of human preference, not human preference itself, it has blind spots, and an unconstrained optimizer will reliably find and exploit those blind spots, a phenomenon called reward hacking (covered in detail below). The KL penalty caps how far the policy is allowed to travel from a known-reasonable starting distribution per unit of reward gained, which keeps the optimization inside the region where the reward model's judgments are actually trustworthy.

PPO specifically is used to perform this optimization because vanilla policy gradient methods are prone to destructively large updates when the reward signal is noisy, which a learned reward model's signal certainly is. PPO clips the policy update so that the probability ratio between the new and old policy for any given action does not move too far in a single step:

L_CLIP(θ) = E_t [ min( ρ_t(θ) A_t,  clip(ρ_t(θ), 1-ε, 1+ε) A_t ) ]

where ρ_t(θ) = π_θ(a_t|s_t) / π_&theta_old;(a_t|s_t)

Here A_t is an advantage estimate (how much better this action was than the policy's average action in that state), typically computed with a separate learned value/critic network trained alongside the policy. This means a full PPO-based RLHF run requires keeping four models resident at once during training: the policy being optimized, a frozen reference copy of the SFT model for the KL term, the reward model, and the value network for the advantage estimate. This is a substantial engineering and memory burden, and it is the main reason RLHF via PPO is considered heavyweight relative to the alternatives discussed next.

For the full reinforcement learning foundations behind this section, including Markov decision processes, the Bellman equations, the full PPO derivation, and GRPO, see the dedicated Reinforcement Learning guide.

Refer: InstructGPT / RLHF paper

Reward Hacking, Precisely

Reward hacking is what happens when the policy finds a way to increase the reward model's score without actually increasing true response quality as a human would judge it. It is not a rare edge case, it is close to an inevitable consequence of optimizing hard against any imperfect, learned proxy objective, a phenomenon closely related to what is sometimes called Goodhart's law: when a measure becomes a target, it stops being a good measure.

Common, well-documented forms of reward hacking in language model RLHF include:

• Length hacking. Reward models trained on human preference data frequently learn a spurious correlation between response length and quality, because annotators, all else equal, mildly prefer longer, more thorough-looking answers. An unconstrained policy will exploit this by producing needlessly long responses that pad out content without adding real information, since length alone increases the reward score.
• Sycophancy. If annotators (even slightly, even unconsciously) rate agreement and validation more highly than correct pushback, the policy learns to agree with the user's stated opinion or claims regardless of whether they are true, because agreement correlates with higher reward in the training data.
• Confident hedging patterns. The policy can learn to attach reward-correlated phrases (disclaimers, caveats, or conversely overconfident framing) that raised scores in the training distribution, independent of whether that framing is actually appropriate for the current response.
• Format gaming. Bullet points, bold headers, or a specific structural template that happened to correlate with higher-rated responses in the RM training data get overproduced regardless of whether that structure suits the current prompt.

Mitigations include: keeping the KL penalty coefficient β large enough to bound policy drift, periodically refreshing the reward model on comparisons sampled from the current (already partially optimized) policy so the RM keeps seeing and correctly scoring whatever new patterns the policy is producing, explicitly normalizing for length in the reward signal, and running held-out human evaluation throughout training (not just automatic reward scores) so that reward-score improvements which do not correspond to real human-judged quality improvements are caught early rather than after deployment.

DPO and the Direct Preference Optimization Family

Direct Preference Optimization (DPO) removes the separate reward model and the RL loop entirely, while provably optimizing the same underlying objective as the KL-constrained RLHF formulation above. The derivation starts from a known result: for the RLHF objective J(θ) defined earlier, the optimal policy has a closed form in terms of the reward function and the reference policy:

π*(y|x) = (1/Z(x)) · π_ref(y|x) · exp( r(x,y) / β )

where Z(x) is a normalizing partition function summed over all possible responses y, which is intractable to compute directly. Rearranging this expression solves for the reward as a function of the optimal policy:

r(x, y) = β log( π*(y|x) / π_ref(y|x) ) + β log Z(x)

The key move in DPO is substituting this reformulation of the reward back into the Bradley-Terry preference loss used for reward modeling. Because the loss only ever involves the difference r(x, y_w) - r(x, y_l) for the same prompt x, and Z(x) depends only on x, the intractable partition function appears identically in both terms and cancels out exactly. What remains is a loss computable directly from the policy's own log-probabilities, with no reward model and no sampling loop needed at all:

L_DPO(θ) = - E_{(x, y_w, y_l)} [
    log σ( β log(π_θ(y_w|x)/π_ref(y_w|x))
              - β log(π_θ(y_l|x)/π_ref(y_l|x)) )
]

In implementation, this means: run both the trainable policy and a frozen reference model (usually a copy of the SFT checkpoint) forward on the preferred and rejected responses, extract sequence log-probabilities under each, and optimize the logistic loss above with plain gradient descent, exactly as you would any supervised objective. There is no sampling from the policy during training, no reward model, and no critic network.

The gradient of this loss has a clean interpretation. Differentiating with respect to the policy's parameters shows that each training example pushes up the log-probability of the preferred response and pushes down the log-probability of the rejected response, with the magnitude of that push scaled by how wrong the model's implicit reward ranking currently is: examples where the policy already strongly prefers the correct response contribute a small gradient, examples where the policy currently has the ranking backwards contribute a large one. This falls directly out of the sigmoid's derivative and is essentially the same self-scaling behavior seen in ordinary cross-entropy.

β plays exactly the role the KL coefficient played in RLHF: it controls how far the policy is permitted to move from the reference model. A small β allows larger, more aggressive shifts toward preferred responses; a large β keeps the policy closer to the reference and produces more conservative updates.

from trl import DPOTrainer

trainer = DPOTrainer(
    model=policy_model,
    ref_model=ref_model,
    train_dataset=preference_pairs,  # prompt, chosen, rejected
    tokenizer=tokenizer,
    beta=0.1,  # KL regularization strength
)
trainer.train()

Because DPO needs only a static, offline preference dataset rather than online sampling and reward evaluation, it is substantially simpler to implement, cheaper to run (no reward model inference during training, no critic network, no PPO clipping machinery), and empirically more stable to train than PPO based RLHF in most reported settings. This has made it the more common default for preference tuning, with PPO style RLHF reserved for situations that genuinely need an online, continuously updated reward signal rather than a fixed offline comparison dataset.

Variants that patch specific DPO weaknesses

• IPO (Identity Preference Optimization). Standard DPO can overfit sharply on preference pairs where the two responses are nearly indistinguishable in quality, since the logistic loss keeps pushing the log-probability gap wider even after the ranking is already correctly learned. IPO replaces the logistic loss with a bounded objective that stops rewarding further increases in the gap once a target margin is reached, which reduces this overfitting tendency.
• KTO (Kahneman-Tversky Optimization). Standard DPO requires paired comparisons (a chosen and a rejected response for the same prompt). KTO is designed to work with unpaired binary feedback instead, meaning each individual response is labeled simply as good or bad in isolation, without needing another response for the same prompt to compare against. This is a large practical advantage when collecting explicit pairwise comparisons is expensive but simple thumbs up or thumbs down signal is already available at scale.
• ORPO (Odds Ratio Preference Optimization). Combines the SFT objective and the preference objective into a single joint training run rather than treating SFT and preference tuning as two fully separate stages, using an odds ratio based penalty term to discourage the rejected response's likelihood while simultaneously training on the preferred response as a standard SFT target. This removes the need for a separate reference model entirely.
• SimPO (Simple Preference Optimization). Removes the reference model dependency from DPO by using the average log-probability per token (rather than the log-probability ratio against a reference policy) as the implicit reward, which also naturally corrects for length bias, since the reward is length normalized by construction rather than being a raw sequence log-probability that favors shorter or longer sequences depending on how the ratio behaves.

The pattern across all of these variants is the same underlying idea expressed with different loss functions: optimize the likelihood of preferred outputs relative to dispreferred ones, under some form of regularization that prevents the policy from drifting arbitrarily far from a sensible starting distribution. The field iterates quickly on the exact loss formula, but this core pattern has remained stable.

Constitutional AI and RLAIF

Human preference labeling is expensive, slow to scale, and can itself be inconsistent or biased toward what looks good rather than what is good. Constitutional AI and the broader category of Reinforcement Learning from AI Feedback (RLAIF) address this by replacing some or all human preference labels with preferences generated by a model, guided by a written set of principles (a "constitution") rather than by a human annotator's individual judgment.

The typical pipeline has two phases. First, a supervised phase: the model is prompted to critique and revise its own outputs against a set of stated principles (for example, prefer responses that avoid harmful advice, or prefer responses that acknowledge uncertainty rather than fabricate confident answers), and the revised outputs become new SFT training data. Second, a preference phase: instead of a human choosing between two candidate responses, a model is prompted with the same constitutional principles and asked to choose which of two candidates better satisfies them, producing an AI generated preference dataset that a reward model (or a DPO style pipeline) can then be trained on directly, exactly as if the labels had come from a human.

The main advantage is scale: AI generated preference labels can be produced far faster and cheaper than human labels, and the constitution provides an explicit, inspectable, editable statement of what the model is being aligned toward, rather than that policy being implicit in whatever an anonymous pool of human annotators happened to prefer. The main risk is that the judging model's own biases and blind spots get baked into the preference data at scale, and any systematic error in how the judge model interprets the constitution propagates directly into the trained policy, potentially with no human check in the loop at all unless deliberately added back through auditing and spot checks.

Preference Data and Red Teaming

The ceiling on alignment quality is set by the quality of the preference data, not by which optimization algorithm is used on top of it. Labelers need clear, specific rubrics covering factuality, helpfulness, safety, and style, because vague instructions like "pick the better response" produce noisy, inconsistent labels that a reward model or DPO run will faithfully learn to reproduce, noise and all. A diverse prompt distribution during data collection matters just as much: if preference pairs are collected mostly from coding prompts, the resulting alignment will generalize poorly to creative writing, medical questions, or ambiguous multi-turn conversations, since the model has never received comparative signal in those regions of the input space.

Red teaming is the practice of proactively and adversarially probing a model before launch to find jailbreaks, biased outputs, and capability or safety gaps, rather than waiting for real users to discover them in production. Red teamers deliberately try prompt injection, roleplay framings designed to elicit unsafe content, incremental escalation across a conversation, and other adversarial techniques. Findings from red teaming feed back into the alignment pipeline in three concrete ways: new preference pairs are constructed specifically targeting the discovered failure mode, system prompts and runtime guardrails are hardened to close the gap quickly without waiting for a full retraining cycle, and dedicated safety classifiers are trained or updated to catch the specific pattern at inference time. Alignment, in practice, is never a one shot training run, it is an iterative loop of probing, labeling, retraining, and re-evaluating that continues for the life of the product.

Evaluating Alignment

Automatic metrics alone are known to be an incomplete and sometimes misleading measure of alignment quality, precisely because of the same reward hacking dynamics discussed earlier: a metric that becomes the optimization target stops reliably measuring the thing it was meant to proxy. Robust evaluation practice combines several complementary signals:

• Human side by side evaluation on prompts sampled from, or representative of, real production traffic, where evaluators directly compare the candidate model's output against a baseline (often the previous production checkpoint) and record a preference, giving a ground truth signal that automatic metrics can be checked against.
• Safety benchmarks measuring toxicity rates, jailbreak success rate against a standard adversarial prompt suite, and refusal accuracy (both that the model refuses genuinely unsafe requests, and that it does not over refuse benign ones, since over refusal is its own failure mode that directly hurts helpfulness).
• Capability retention checks, meaning standard benchmarks for math, coding, reasoning, and factual knowledge, run before and after each alignment stage, specifically to detect the alignment tax described below.
• LLM as judge evaluation, where a separate (usually stronger) model scores or compares outputs at scale, which is far cheaper and faster than human evaluation but requires careful calibration against real human judgments, since judge models are themselves known to drift over time and to systematically favor longer, more verbose, more confidently phrased responses independent of actual quality, echoing the same length and confidence biases seen in human annotated reward models.

Alignment tax refers to the measurable loss of raw capability (most commonly seen on math, coding, and other tasks with a single objectively correct answer) that can occur after heavy safety or preference tuning, as a side effect of the policy shifting away from its pretrained and SFT distribution to satisfy the preference or safety objective. It is mitigated by mixing general capability data into the alignment training set rather than training purely on preference or safety data in isolation, keeping KL penalties (or their DPO equivalent, the β coefficient) conservative enough to bound how far the policy drifts, and gating any new checkpoint behind capability regression tests before it is allowed to ship to production, exactly the same discipline used to catch catastrophic forgetting during ordinary fine-tuning.

System Prompts vs Weight Level Alignment

There are two distinct layers at which a model's behavior can be shaped, and production systems typically use both together rather than relying on either alone.

System prompts and runtime guardrails (input and output classifiers, allow or deny lists for specific tool calls, retrieval filtering) shape behavior at inference time without touching the model's weights at all. Their major advantage is speed and reversibility: a policy change can be deployed in minutes by editing a prompt or a classifier threshold, with no retraining required, which is essential for reacting quickly to a newly discovered failure mode found through red teaming or live production incidents.

Weight level alignment, meaning the behavior actually instilled through SFT, RLHF, DPO, or constitutional methods, generalizes more robustly to adversarial or unusual inputs than a system prompt does, and it does not depend on the system prompt remaining secret or intact. A behavior that exists only because of a system prompt instruction can, in many cases, be argued around, ignored, or overridden by a sufficiently adversarial user turn, whereas a behavior baked into the weights through training on many diverse examples of the desired behavior tends to hold up across a much wider range of inputs the model was never explicitly instructed about at inference time.

In practice, production systems layer these together deliberately: weight level alignment establishes the robust, generalizing foundation for what the model does and refuses by default, while system prompts and runtime policies handle product specific rules, tool use constraints, and dynamically injected context (retrieved documents, user preferences, session state) that legitimately change from deployment to deployment and cannot reasonably be baked into a single shared set of weights.