Fine-Tuning

Purpose of Fine-Tuning

Pretrained base models predict continuation. Fine-tuning updates weights so the model follows instructions, speaks in a desired tone, excels at a domain, or emits structured outputs. It is cheaper and faster than pretraining because datasets are smaller and runs are shorter, but it can still shift model behavior dramatically.

Common goals include customer support tone, legal document analysis, code generation in internal APIs, JSON-only responses, and multilingual support for underrepresented languages in the base model.

Supervised Fine-Tuning (SFT)

SFT trains on labeled input-output pairs: prompts (instructions plus context) and target completions. Loss is computed only on the target tokens, not the prompt, so the model learns to produce answers, not memorize prompts.

# Mask prompt tokens with -100 so loss applies only to the answer
labels = input_ids.clone()
labels[:prompt_len] = -100

outputs = model(input_ids=input_ids, labels=labels)
loss = outputs.loss  # cross-entropy on answer tokens only

Dataset design drives results:

• Cover edge cases and refusal scenarios, not only happy paths
• Match production prompt format and chat template exactly
• Balance task types so the model does not overfit one pattern
• Keep epochs low (often 1-3) to avoid catastrophic forgetting of general capability

Full Fine-Tuning vs Parameter-Efficient Methods

Full fine-tuning

All weights update. Maximum flexibility, highest memory and storage cost, greatest risk of forgetting general knowledge on small datasets.

LoRA (Low-Rank Adaptation)

Inserts trainable low-rank matrices into attention and sometimes MLP layers while freezing the base weights. A 70B model may train only tens of millions of adapter parameters. Multiple LoRA adapters can swap at serving time for multi-tenant products.

# W' = W + A·B   (A: d×r, B: r×d, rank r ≪ d)
# Only A and B are trained; base weight W stays frozen

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()  # ~0.1% of weights trainable

QLoRA

Quantizes the frozen base model to 4-bit (NF4) and trains LoRA adapters in BF16/FP16. Enables large-model fine-tuning on limited GPU memory with modest quality tradeoffs.

Adapters and prompt tuning

Smaller bottleneck layers or learned soft prompts at the input provide even lighter touch adaptation when full LoRA is unnecessary.

Continued Pretraining vs Instruction Tuning

Continued pretraining (domain-adaptive pretraining) feeds raw domain text with next-token loss to inject vocabulary and facts before SFT. Useful for medicine, finance, or internal wiki corpora where the base model lacks terminology.

Instruction tuning comes after and teaches task format. Order matters: domain knowledge first, behavior shaping second. Skipping straight to SFT on niche tasks without domain exposure may yield fluent but factually weak outputs.

Hyperparameters and Overfitting

• Learning rate - typically 10x to 100x lower than pretraining
• Rank (LoRA) - higher rank captures more capacity but risks overfitting
• Batch size and sequence length - match production limits
• Evaluation - hold out real user-style prompts, not training paraphrases

Signs of overfitting: near-perfect training loss, degraded general benchmarks, repetitive phrasing, and sensitivity to minor prompt wording changes.

Serving Fine-Tuned Models

LoRA adapters can load atop a shared base in vLLM, TGI, or custom servers, reducing duplicate weight storage. Version adapters independently from the base checkpoint. Document which base revision each adapter was trained on; base upgrades may require re-tuning.

For API providers, fine-tuning APIs abstract storage and training, but you still own dataset quality, eval, and rollback strategy when a new fine-tune regresses production metrics.