Fine-Tuning

Adapting pretrained models to specific tasks, domains, and instruction-following behavior.

What Fine-Tuning Actually Changes

A pretrained base model is the result of next-token prediction over a huge, broad text corpus. Its objective during pretraining is purely distributional: given a sequence of tokens, predict a probability distribution over the next token that matches the statistics of the training corpus. Nothing in that objective rewards the model for being helpful, safe, concise, or good at following an instruction — it only rewards matching the corpus distribution. A base model given "Explain quantum entanglement" is just as likely to continue with a list of textbook chapter titles, a forum reply, or a homework assignment as it is to actually explain the concept, because all of those are plausible continuations of that string somewhere in the training data.

Fine-tuning is the process of taking the pretrained weights θ0 and further updating them, usually with a much smaller and much more curated dataset, so that the resulting distribution pθ(y|x) concentrates probability mass on the kind of outputs you actually want for the inputs you actually care about. It is the mechanism by which "a model that predicts plausible text" becomes "a model that answers questions," "a model that writes SQL," "a model that refuses unsafe requests," or "a model that speaks in your company's tone."

Three properties distinguish fine-tuning from pretraining in practice:

The Cross-Entropy Objective, Precisely

Every fine-tuning method discussed here (SFT, LoRA, QLoRA, continued pretraining) shares the same underlying loss: autoregressive cross-entropy, also called the negative log-likelihood of the target sequence under the model.

For a sequence of tokens y = (y1, y2, ..., yT), the model factorizes the joint probability autoregressively:

p_θ(y) = ∏_{t=1}^{T} p_θ(y_t | y_{<t})

Training minimizes the negative log-likelihood, summed (or averaged) over target tokens:

L(θ) = - (1/T) ∑_{t=1}^{T} log p_θ(y_t | y_{<t})

In practice this is implemented as token-level cross-entropy between the model's predicted logits and the one-hot ground truth token, computed by a softmax followed by a negative log:

logits = model(input_ids)                     # (batch, seq_len, vocab_size)
log_probs = log_softmax(logits, dim=-1)
loss = -log_probs.gather(-1, targets.unsqueeze(-1)).squeeze(-1)
loss = loss.mean()  # over non-masked positions only

The gradient of cross-entropy with respect to the logits has an unusually clean form. If z is the logit vector, p = softmax(z), and y is the one-hot target, then:

∂L/∂z = p - y

This means the gradient signal at every token is simply "predicted probability minus 1 at the correct token, predicted probability at every other token." Confident, correct predictions produce near-zero gradient; confident, wrong predictions produce large gradient in the direction of the correct token. This is why cross-entropy is well-behaved for optimization — the gradient magnitude is self-scaling to how wrong the model currently is.

Supervised Fine-Tuning (SFT)

SFT trains on labeled pairs: a prompt (system instruction, conversation history, retrieved context, tool outputs — whatever conditioning the model needs) and a target completion written or curated by a human, another model, or a hybrid pipeline. The critical implementation detail is loss masking: the cross-entropy loss is computed only over the tokens belonging to the target completion, never over the prompt tokens.

Why mask the prompt? Two reasons. First, you do not want the model to spend capacity learning to predict the prompt itself — the prompt is given at inference time, not generated, so there is nothing to learn there and training on it wastes gradient signal and can bias the model toward regurgitating prompt-like text. Second, if the same prompt appears with different valid completions across the dataset (a plausible situation with multi-turn dialogue or paraphrased instructions), training on the prompt tokens injects noisy, non-generalizable gradient into the model.

# Mask prompt tokens with -100 (PyTorch's ignore_index for CrossEntropyLoss)
input_ids = tokenizer.encode(prompt + completion)
prompt_len = len(tokenizer.encode(prompt))

labels = input_ids.clone()
labels[:prompt_len] = -100     # loss ignores these positions

outputs = model(input_ids=input_ids, labels=labels)
loss = outputs.loss            # cross-entropy on completion tokens only

In multi-turn conversational data, the same masking principle extends per-turn: every assistant turn contributes to the loss, every user/system/tool turn is masked out. This is what "training on assistant turns only" means in most SFT pipelines.

Chat templates and why exact formatting is non-negotiable

Modern instruction-tuned models are trained with a strict token-level template that wraps roles (system/user/assistant/tool) in special tokens, for example something structurally like:

<|system|>
You are a helpful assistant.
<|user|>
What is the capital of France?
<|assistant|>
The capital of France is Paris.
<|end|>

If your fine-tuning data does not exactly reproduce the special tokens, spacing, and role markers the model was originally instruction-tuned with (or, if training a base model from scratch into a chat model, the template you intend to use at serving time), you introduce a train/inference mismatch. The model will have learned statistics conditioned on one template and be queried with another, which degrades quality unpredictably — sometimes catastrophically, sometimes subtly (worse instruction-following, ignored system prompts, broken stop-token behavior so generation never terminates cleanly).

Dataset design principles

Where SFT data comes from

Three common sources, usually combined: (1) human-written demonstrations, highest quality but expensive and slow to scale; (2) distillation from a stronger model, where a large "teacher" model generates completions that a smaller "student" model is trained to imitate — cheap and scalable but caps the student's quality at (and typically below) the teacher's, and can propagate the teacher's errors and biases; (3) self-instruct / synthetic pipelines, where a model generates its own instruction-completion pairs from a small seed set, followed by filtering for quality, diversity, and correctness (often using a separate model or rule-based verifier as a judge).

Full Fine-Tuning vs Parameter-Efficient Fine-Tuning (PEFT)

Full fine-tuning

Every parameter in the model is unfrozen and updated by gradient descent. This gives maximum representational flexibility — there is no constraint on which parameters can move or by how much — but it has three costs that dominate practical decision-making:

LoRA (Low-Rank Adaptation) — full derivation

LoRA is motivated by the empirical observation that the weight update needed to adapt a large pretrained model to a new task tends to have low "intrinsic rank" — that is, the update matrix, while formally full-rank-sized, can be well-approximated by a much lower-rank matrix without significant loss of adaptation quality.

Take any weight matrix in the network that is normally updated during fine-tuning, say a projection matrix W ∈ R^{d×k} in an attention block (e.g. the query or value projection). Full fine-tuning would learn an additive update ΔW ∈ R^{d×k} so that the effective weight becomes W + ΔW. LoRA instead freezes W entirely and constrains the update to a low-rank product:

ΔW = B A,        where B ∈ R^{d×r},  A ∈ R^{r×k},  r « min(d, k)

The forward pass becomes:

h = W x + ΔW x = W x + B A x = W x + B(Ax)

Only A and B receive gradients; W is frozen and contributes no gradient computation or optimizer state at all. Because r is typically 4-64 while d and k are often in the thousands, the number of trainable parameters per adapted matrix is r(d+k) instead of d×k — often a reduction of two to three orders of magnitude.

Initialization matters. A is initialized with small random values (e.g. Gaussian scaled by 1/r or Kaiming initialization), while B is initialized to exactly zero. This guarantees ΔW = BA = 0 at the very start of training, so the adapted model is numerically identical to the base model at step zero, and training begins from a known-good point rather than an arbitrarily perturbed one.

The alpha/rank scaling factor. LoRA introduces a scaling constant, usually written α, so the effective update is:

ΔW = (α/r) B A

This decouples the "step size" of the adaptation from the rank you choose. Without this scaling, increasing r would implicitly change the effective learning rate applied to ΔW, confounding the two hyperparameters. A common convention is α = 2r, but this is tuned per task like any other hyperparameter.

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()  # commonly ~0.05-0.5% of total weights

Which matrices to target matters a lot. Applying LoRA only to the query and value projections (the original paper's default) is cheap but limits capacity. Applying it to all attention projections (q_proj, k_proj, v_proj, o_proj) plus the MLP/feed-forward projections (gate_proj, up_proj, down_proj in a SwiGLU-style block) captures substantially more adaptation capacity, since a large fraction of a transformer's parameters — and a large fraction of where task-specific computation actually happens — live in the MLP blocks, not just attention.

Merging at inference. Because ΔW and W have the same shape, they can be summed once after training — W' = W + (α/r)BA — producing a single dense weight matrix with zero additional inference latency and no extra parameters at serving time. Alternatively, the adapter can be kept separate and swapped in dynamically, trading a small amount of extra matmul compute for the ability to serve many task-specific adapters on top of one shared base model in memory.

QLoRA — fine-tuning under severe memory constraints

QLoRA combines LoRA with aggressive quantization of the frozen base model, enabling fine-tuning of very large models on a single consumer or prosumer GPU. Three techniques stack together:

Critically, only the LoRA adapters (A and B) are trained in higher precision (bf16/fp16); the frozen base weights stay in 4-bit for the entire forward and backward pass, with on-the-fly dequantization to a computation-friendly precision only at the moment they're multiplied. Gradients never flow into the quantized base weights at all — they are frozen by construction of the LoRA formulation, so quantization error in the frozen weights degrades starting behavior slightly, but never introduces the exploding/vanishing gradient issues that naive low-bit training would otherwise risk.

The net effect: a model that would require roughly 140GB+ of memory just for full-precision weights (a ~70B model in fp16, i.e. ~2 bytes/parameter) can be loaded for QLoRA fine-tuning in roughly a quarter of that (4-bit weights, i.e. ~0.5 bytes/parameter plus small overhead), with only a small, well-documented quality gap versus full-precision LoRA, and effectively no gap versus full fine-tuning on most benchmarks reported in the original QLoRA work.

Other PEFT families

Practical default: Start with LoRA or QLoRA (rank 8-64, applied to attention and MLP projections) on an instruction-tuned checkpoint. Move to full fine-tuning only when evaluation on held-out data shows the adapter's capacity, not the dataset or hyperparameters, is the actual bottleneck — this is a real but relatively rare situation in practice, most commonly seen when adapting to a genuinely new domain vocabulary or a very large, diverse instruction-tuning dataset.

Continued Pretraining vs Instruction Tuning

Continued pretraining, also called domain-adaptive pretraining (DAPT), applies the exact same next-token-prediction objective used in original pretraining, but on a narrower, domain-specific unlabeled corpus (e.g. millions of tokens of legal filings, medical literature, or an internal company wiki) rather than labeled prompt-completion pairs. There is no prompt/completion split and no loss masking — every token in the corpus contributes to the loss, exactly as in pretraining, just with a smaller and more targeted dataset and (usually) fewer total training steps.

The purpose is to shift the model's underlying knowledge and vocabulary before you ever try to teach it a task format. A base model that has never seen much biomedical text will not reliably know that "MI" usually means myocardial infarction in a clinical note, regardless of how well-formatted your instruction-tuning examples are — the token-level statistical associations simply were never learned. Continued pretraining injects that domain knowledge directly.

Order matters. The standard recipe is: (1) continued pretraining on raw domain text to build vocabulary and factual grounding, then (2) instruction tuning (SFT) on labeled examples to teach task format, tone, and instruction-following behavior on top of that domain knowledge. Reversing the order, or skipping continued pretraining entirely for a genuinely unfamiliar domain, tends to produce a model that is fluent and confident in its output format but frequently wrong or vague on domain facts — the instruction-tuning stage teaches the model how to answer, not what the true answer is, and it can only draw on knowledge that's actually present in its weights from an earlier stage.

A practical risk with continued pretraining: because it uses the full next-token objective over a large corpus, it is much more prone to catastrophic forgetting of general capability than SFT is, simply because more total gradient steps are typically taken over more tokens. Mitigations include mixing in a fraction (commonly 5-20%) of general-domain text alongside the domain-specific corpus, and using a low learning rate with a short warmup and decay schedule rather than the full pretraining schedule.

Beyond SFT: Preference Optimization (RLHF and DPO)

SFT teaches a model to imitate example completions, but imitation alone has a ceiling: it can only be as good as the completions in the dataset, and it provides no signal about relative quality between two plausible-but-different answers. Preference optimization methods address this by training on comparisons — "response A is better than response B for this prompt" — rather than single demonstrations.

RLHF (Reinforcement Learning from Human Feedback)

The classical RLHF pipeline has three stages:

  1. SFT stage — as described above, producing an initial instruction-following policy πSFT.
  2. Reward modeling — human annotators are shown pairs of completions (yw, yl) for the same prompt x ("w" for the preferred/winning response, "l" for the rejected/losing one), and a separate reward model rφ(x, y) is trained to score responses such that the preferred one scores higher. The standard loss is derived from the Bradley-Terry model of pairwise preference: 
    L(φ) = - E_{(x,y_w,y_l)} [ log σ( r_φ(x, y_w) - r_φ(x, y_l) ) ]
    where σ is the logistic sigmoid. This trains the reward model to assign a higher scalar score to whichever response the human preferred, without needing an absolute quality scale — only relative ordering.
  3. RL fine-tuning (typically PPO) — the policy is further updated to maximize the learned reward, using Proximal Policy Optimization, with a KL-divergence penalty against the SFT policy to keep the model from drifting too far and degenerately exploiting the reward model: 
    objective(θ) = E_{x, y~π_θ} [ r_φ(x, y) ] - β · KL( π_θ(·|x) ‖ π_SFT(·|x) )
    The KL term is essential: reward models are imperfect proxies for true human preference, and a policy optimized against them without constraint will find "reward hacks" — outputs that score highly on the learned reward model but are not actually good by the standard a human would apply (excessive hedging, sycophancy, empty verbosity, listing disclaimers, etc., depending on what spurious pattern correlates with high reward in the training data). The KL penalty, controlled by β, bounds how far the policy can move from the known-reasonable SFT starting point per unit of reward gained.

RLHF via PPO is notoriously complex to implement correctly and unstable to tune: it requires keeping four models in memory simultaneously during training (the policy being trained, a frozen reference copy for the KL term, the reward model, and a value/critic network for the PPO advantage estimate), and small hyperparameter mistakes (KL coefficient, reward normalization, learning rate, batch composition) can cause reward hacking or training collapse.

DPO (Direct Preference Optimization)

DPO removes the separate reward model and the RL loop entirely, while provably optimizing the same objective as RLHF under the KL-constrained reward maximization formulation above. The key mathematical insight is that the optimal policy for the RLHF objective has a closed-form relationship to the reward function:

r(x, y) = β log( π_θ(y|x) / π_ref(y|x) ) + β log Z(x)

where Z(x) is a partition function that depends only on the prompt x, not on the response y. Substituting this reformulation of the reward directly into the Bradley-Terry preference loss causes the intractable partition function Z(x) to cancel out algebraically (because it appears identically in both the winning and losing response's reward term, and the loss only depends on their difference), leaving a loss that can be computed directly from the policy's own log-probabilities, with no reward model and no sampling/RL loop required:

L_DPO(θ) = - E_{(x,y_w,y_l)} [
    log σ( β log(π_θ(y_w|x)/π_ref(y_w|x))
              - β log(π_θ(y_l|x)/π_ref(y_l|x)) )
]

In implementation terms: run both the trainable policy and a frozen reference model (usually initialized as a copy of the SFT checkpoint) forward on the same preferred/rejected pair, take the sequence log-probabilities under each, and optimize the above logistic loss with ordinary supervised-style gradient descent — no sampling from the policy during training, no reward model, no critic network, no PPO instabilities. The gradient of this loss has an intuitive interpretation: it increases the log-probability of the preferred response and decreases the log-probability of the rejected response, with the update weighted more heavily when the current implicit reward model has the ranking wrong (i.e. when the model currently prefers the rejected response), which falls directly out of the sigmoid's derivative.

β plays the same role as in RLHF's KL penalty: it controls how far the policy is allowed to deviate from the reference model. Small β allows more aggressive movement toward the preferred responses (larger effective step size relative to the reference), large β keeps the policy closer to the reference.

Because DPO only requires a static, offline preference dataset rather than online sampling and reward evaluation, it is dramatically simpler to implement, cheaper to run, and more stable to train than PPO-based RLHF, and it has become the more common default for preference tuning in practice, with PPO-style RLHF (or its variants) reserved for settings where truly online, adaptive reward signals are needed.

Hyperparameters, in Depth

Diagnosing overfitting

The clearest quantitative signature is a widening gap between training loss (which keeps falling) and validation loss (which plateaus, then rises). Qualitatively, an overfit model shows: near-verbatim reproduction of training examples when given prompts similar to training prompts; degraded performance on general-capability benchmarks the model previously scored well on (a direct measurement of catastrophic forgetting); repetitive or templated phrasing that mirrors patterns overrepresented in the fine-tuning set; and disproportionate sensitivity to small, semantically irrelevant wording changes in the prompt (a sign the model has learned surface statistical correlations specific to the training phrasing rather than the underlying task).

Catastrophic Forgetting — Mechanism and Mitigation

Catastrophic forgetting refers to a fine-tuned model losing capability on tasks or knowledge it demonstrated well before fine-tuning, as a side effect of weight updates optimized purely for the new, narrower objective. Mechanistically, gradient descent has no built-in notion of "don't break what already works" — every update step is computed only from the current batch's loss, and a parameter that happens to be important for some pretrained capability but irrelevant to the fine-tuning task can still be pushed away from its useful value if doing so reduces the fine-tuning loss, even slightly.

Several factors increase forgetting risk: high learning rates (larger steps away from the pretrained optimum), many epochs over a narrow dataset (repeated pressure in the same direction), full fine-tuning versus PEFT (more parameters free to move), and a fine-tuning distribution very different from, or much narrower than, the pretraining distribution (concentrated gradient pressure on a specific sub-region of parameter space).

Common mitigation strategies:

Serving Fine-Tuned Models

LoRA adapters, being small and structurally separate from the frozen base weights until merged, can be loaded on top of a single shared base model at serving time — engines such as vLLM and Hugging Face TGI support dynamically swapping or even simultaneously serving multiple LoRA adapters against one resident base model in GPU memory, which is the foundation of most multi-tenant, multi-task fine-tuned deployment setups: one base model deployment, many lightweight task- or customer-specific adapters loaded per request.

Two serving strategies for LoRA specifically:

Regardless of strategy, version adapters independently from the base checkpoint and document precisely which base model revision each adapter was trained against — an adapter trained against one base checkpoint is not guaranteed to behave correctly, or even to be numerically well-formed, if merged onto or served alongside a different (even seemingly minor) revision of the base weights, since the low-rank update was learned relative to that specific frozen weight matrix.

For hosted fine-tuning APIs (where a provider abstracts away the training infrastructure), the provider typically owns the training loop, hardware, and checkpoint storage, but you still own dataset quality, evaluation methodology, and rollback strategy — if a new fine-tune regresses a production metric after deployment, you need your own held-out evaluation suite and a fast rollback path to the previous checkpoint or adapter version, independent of whatever the provider offers.