Interview: Pretraining

Next-token prediction (causal language modeling): given all previous tokens, predict the probability distribution over the next token, trained with cross-entropy loss against the actual next token in the corpus. This simple, self-supervised objective is what lets pretraining scale to trillions of tokens without any human labeling.

• Deduplication - near-duplicate documents/sequences hurt generalization and can cause memorization.
• Quality filtering - heuristics or classifiers to remove boilerplate, spam, and low-quality web text.
• Toxicity/PII filtering - reduce harmful content and personal data leakage.
• Domain mixing - deliberate ratios of web text, code, books, academic papers, multilingual data - the mix strongly affects downstream capability.
• Decontamination - removing benchmark test data from training data to avoid inflated eval scores.

Scaling laws describe how loss improves predictably as you scale model parameters (N), training tokens (D), and compute (C). The Chinchilla finding was that most earlier models (like GPT-3) were significantly under-trained relative to their size - for a fixed compute budget, loss is minimized by scaling parameters and training tokens in roughly equal proportion (~20 tokens per parameter), rather than making models bigger without giving them enough data. This reshaped how labs allocate compute budgets.

Training compute is a one-time cost proportional to ~6ND (params × tokens), while inference compute is a recurring per-request cost proportional to model size and output length, paid every time the model is used. This is why labs increasingly train smaller models on far more tokens than "Chinchilla-optimal" (e.g. LLaMA-3) - a slightly more expensive training run is worth it if it produces a smaller, cheaper-to-serve model for the same quality.

Mixed-precision training keeps most computation in a lower-precision format (16-bit) for speed and memory savings while keeping a master copy of weights and certain reductions in fp32 for numerical stability. bf16 (bfloat16) has the same exponent range as fp32 (just less mantissa precision), so it's far less prone to overflow/underflow than fp16, avoiding the need for loss scaling - which is why it's the default for LLM pretraining on modern GPUs/TPUs.

Loss spikes can come from bad data batches, numerical instability in attention/optimizer states, or learning rate being too high for the current curvature. Mitigations include: gradient clipping, restarting training from a checkpoint before the spike with a different data order or lower LR, using more stable normalization (RMSNorm, QK-norm), and z-loss regularization on logits.

Training starts with a small LR that gradually increases (warmup) to avoid destabilizing large random initial gradients, then follows a decay schedule (cosine or linear) down toward a small value as training progresses, letting the model make coarse progress early and fine-tune its loss landscape later without overshooting minima.

Continual pretraining resumes base LM training on new domain data (code, medicine, a new language) before task-specific fine-tuning. Mid-training is a longer adaptation phase between general pretraining and SFT, often on higher-quality curated data. It injects domain knowledge without full fine-tuning on downstream tasks.

Z-loss adds a penalty on the log-sum-exp of logits to discourage extreme logit magnitudes that cause training instability and loss spikes. It is a cheap regularizer used in large-scale runs (e.g. PaLM) to keep softmax numerically stable over long training.