Pretraining

Building base model capability from raw corpora through next-token prediction at scale — architecture, data, optimization, distributed systems, and scaling laws in full technical depth.

1. Objective and Mathematical Formulation

Pretraining fits an autoregressive language model to a next-token prediction objective. Given a token sequence t1, t2, ..., tn drawn from a corpus, the model parameterizes a conditional distribution Pθ(ti | t1...ti-1) and is trained to maximize the likelihood of the true continuation. The loss minimized is the average cross-entropy over the sequence:

L(θ) = -(1/N) ∑_{i=1}^{N} log P_θ(t_i | t_1 ... t_{i-1})

This is equivalent to minimizing the Kullback-Leibler divergence between the empirical data distribution and the model's distribution, and it is reported in practice as perplexity, PPL = exp(L), the exponential of the average cross-entropy in nats (or 2L if bits). Lower perplexity means the model assigns higher probability mass to the true next token on average. Perplexity is corpus-dependent and not directly comparable across tokenizers or domains, which is a common evaluation pitfall.

Because every token in a sequence supplies a training signal (teacher forcing), a single forward pass produces N loss terms per sequence of length N, which is what makes next-token prediction so sample-efficient compared to objectives that only supervise a final output. This density of supervision, combined with the near-unlimited availability of unlabeled text, is the central reason self-supervised pretraining scales better than any purely supervised alternative.

Pretraining is distinct from post-training in the objective, not just the data: pretraining has no notion of "correct answer" or "preference" — it purely models the statistics of the training distribution. This means a well-trained base model is a compressed simulator of its training corpus, not an agent aligned to a goal. All the instruction-following, refusal, and persona behavior seen in a deployed assistant is layered on afterward via supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), or similar post-training procedures; pretraining sets the ceiling on world knowledge, linguistic competence, and latent reasoning circuitry that those later stages can only elicit and shape, not create from nothing.

2. Tokenization

Text must be converted to discrete integer IDs before it enters the model. The tokenizer is fixed before pretraining begins because the embedding matrix and output (unembedding) matrix are indexed by vocabulary ID; changing the tokenizer after training requires retraining these layers, at minimum, and often the full model.

2.1 Subword algorithms

2.2 Vocabulary size trade-offs

Larger vocabularies (100k–250k tokens) shorten average sequence length per unit of text, which reduces the number of forward passes needed to cover a fixed amount of information and improves compression, especially for multilingual and code corpora. The cost is a larger embedding and unembedding matrix (parameters scale as vocab_size × d_model for each), more memory for the softmax over the vocabulary, and sparser gradient updates per rare token. Smaller vocabularies (30k–50k) do the opposite: cheaper output layer, but longer sequences for the same text, which increases compute for a fixed context window in tokens.

2.3 Practical considerations

3. Model Architecture

Nearly all modern pretrained language models are decoder-only Transformers using causal (autoregressive) self-attention, but the specific sub-layer choices vary and materially affect training stability, throughput, and downstream quality.

3.1 Causal self-attention

For each layer, input representations X ∈ Rn×d are projected into queries Q, keys K, and values V. Attention weights are computed as:

Attention(Q, K, V) = softmax( (Q K^T) / sqrt(d_k) + M ) V

where M is a causal mask (upper-triangular set to -∞) that prevents position i from attending to positions j > i, preserving the autoregressive property required for next-token prediction. Multi-head attention runs h parallel attention computations with independently learned projections of size d_k = d_model / h, then concatenates and linearly projects the result, allowing different heads to specialize in different relational patterns (syntax, coreference, positional offsets, induction).

3.2 Attention variants for efficiency

3.3 Positional information

3.4 Normalization

Transformers normalize activations to stabilize gradient scale across depth. LayerNorm normalizes each activation vector to zero mean and unit variance, then applies a learned scale and shift. RMSNorm simplifies this by normalizing only by the root-mean-square (dropping mean-centering), which is cheaper and empirically matches or exceeds LayerNorm for large Transformers. Placement matters as much as the formula:

3.5 Feed-forward sub-layer and activation functions

Each Transformer block includes a position-wise MLP, typically expanding d_model to 4× (or ~2.67× for gated variants) and projecting back down. Activation choice affects both quality and compute:

3.6 Mixture-of-Experts (MoE)

Sparse MoE layers replace a single dense MLP with a bank of E expert MLPs and a lightweight router that selects the top-k experts (commonly k=1 or k=2) per token. Only the selected experts execute for that token, so the number of active (compute) parameters per token can be far smaller than the total parameter count — decoupling "knowledge capacity" (total parameters) from "compute cost" (active parameters). Key engineering challenges are load balancing (auxiliary losses that penalize routing collapse onto a few experts), routing instability during early training, and the all-to-all communication cost of dispatching tokens to experts across devices in a distributed setup. MoE models can match the quality of a dense model with far fewer FLOPs per token, or exceed a dense model's quality at matched FLOPs by using far more total parameters.

3.7 KV cache and inference-time architecture consequences

Although this is technically an inference concern, architectural choices at pretraining time (number of KV heads, head dimension, number of layers) directly determine the size of the key/value cache during autoregressive decoding, which scales as 2 × layers × kv_heads × head_dim × sequence_length × batch_size × bytes_per_element. This is why GQA/MQA are chosen at pretraining time rather than patched in afterward.

4. Data Pipeline

Data quality and composition are the dominant lever on downstream model quality, often more impactful than architectural tweaks at fixed compute. A production-grade pretraining pipeline includes the following stages, typically run at petabyte scale on distributed compute (Spark, Ray, or custom MapReduce-style systems).

4.1 Sourcing

4.2 Text extraction and cleaning

Raw crawl data is HTML/PDF/markup that must be converted to clean text: boilerplate removal (navigation, ads, cookie banners), language identification and filtering, encoding normalization, and removal of malformed or non-linguistic content (e.g., minified JS embedded in a page).

4.3 Deduplication

Deduplication matters because repeated content wastes compute (the model re-learns the same signal) and increases verbatim memorization risk, which is both a privacy/copyright concern and a source of contamination when repeated spans overlap with evaluation benchmarks.

4.4 Quality filtering

4.5 Safety and compliance filtering

4.6 Domain mixing and data weighting

The final training mixture is not simply "all filtered data concatenated" — teams tune sampling ratios across domains (web, books, code, academic, dialogue, multilingual) because naive proportional-to-availability mixing over-represents low-value web text and under-represents scarce high-value sources like code or academic writing. Techniques include manually tuned static ratios validated by small-scale ablation runs, and more automated approaches (e.g., DoReMi-style methods) that learn domain weights by optimizing worst-case or average loss across a reference set of domains. Upweighting (repeating) a high-quality but small domain for multiple epochs is common and, up to a few epochs, has been shown empirically not to strongly harm generalization, in contrast to the older assumption that any repetition is harmful.

4.7 Decontamination

Benchmark contamination — evaluation-set text leaking into the training corpus — silently inflates reported benchmark scores without reflecting real capability. Pipelines run n-gram overlap search (typically 13-gram or longer exact matches) between the training corpus and the full suite of intended evaluation benchmarks, removing or flagging matching documents. This must be re-run whenever new benchmarks are adopted, and is complicated by benchmark answers propagating into web text (blogs discussing benchmark questions, leaderboard writeups quoting test items) well after the benchmark's official release.

4.8 Document packing and sequence construction

Because documents vary widely in length while the model consumes fixed-length training sequences, documents are typically concatenated with an explicit end-of-document token and packed into fixed-length blocks (e.g., 4096 or 8192 tokens) to maximize GPU utilization, rather than padding every sequence to the same length individually. Attention masking or position-ID resets at document boundaries can optionally prevent cross-document attention leakage within a packed sequence, trading a small amount of implementation complexity for cleaner document-boundary semantics.

5. Training Configuration and Optimization

5.1 Optimizer

AdamW (Adam with decoupled weight decay) is the near-universal optimizer for large language model pretraining. It maintains per-parameter first-moment (mean) and second-moment (variance) estimates of the gradient:

m_t = β1 m_{t-1} + (1-β1) g_t
v_t = β2 v_{t-1} + (1-β2) g_t^2
θ_t = θ_{t-1} - η ( m_t_hat / (sqrt(v_t_hat) + ε) + λ θ_{t-1} )

Typical settings for large-scale runs: β1 ≈ 0.9, β2 ≈ 0.95 (lower than the default 0.999, which improves stability for LLM training by making the optimizer more responsive to recent gradient statistics), ε ≈ 1e-8 to 1e-5, and weight decay λ ≈ 0.1 applied only to matrix (non-bias, non-norm) parameters. AdamW's memory footprint (two extra states per parameter, in FP32) is itself a major systems constraint at scale — for a model with P parameters, optimizer state alone requires roughly 8P bytes in naive FP32 Adam, which is why memory-efficient variants (8-bit Adam, Adafactor's factored second moments, or sharded optimizer states) are used at the largest scales.

5.2 Learning rate schedule

A short linear (or sometimes exponential) warmup phase (hundreds to a few thousand steps) ramps the learning rate from near zero to its peak value, preventing early large, poorly-conditioned gradients from destabilizing training before the Adam moment estimates have "burned in." After warmup, the rate decays — most commonly following a cosine schedule down to some fraction (often 10%) of the peak by the end of training, though linear decay and, increasingly, warmup-stable-decay schedules (a long constant plateau followed by a short decay, which allows flexible training-length extension without re-deriving the whole schedule) are also used. Peak learning rate itself typically scales down as model size increases, following rough empirical rules of thumb refined per model family.

5.3 Batch size

Batch size (in tokens) is chosen close to the largest size that keeps training stable and matches the "critical batch size" — beyond this point, larger batches yield diminishing returns in loss-per-token processed even though they improve hardware utilization. Many large runs use batch size warmup, ramping from a small batch to the target batch over the first portion of training, which has been observed to further stabilize early optimization. Achieving very large effective batch sizes across many devices without exceeding per-device memory is done via gradient accumulation — running several forward/backward passes with smaller micro-batches and summing gradients before a single optimizer step.

5.4 Regularization and stabilization

5.5 Mixed precision and numerics

Training in full FP32 throughout is prohibitively slow and memory-heavy at scale, so large runs use mixed precision:

6. Distributed Training and Infrastructure

A frontier pretraining run does not fit on a single accelerator, either in parameter memory, optimizer-state memory, or compute time, so multiple forms of parallelism are combined.

6.1 Data parallelism

The simplest form: an identical model replica is placed on each device, each replica processes a different micro-batch, and gradients are averaged across replicas (via all-reduce) before every optimizer step. This scales throughput near-linearly with device count until communication bandwidth becomes the bottleneck, but naive data parallelism requires each device to hold a full copy of parameters, gradients, and optimizer state, which becomes infeasible for models with tens or hundreds of billions of parameters.

6.2 Tensor (model) parallelism

Individual weight matrices are sharded across devices — e.g., splitting an MLP's up-projection column-wise and its down-projection row-wise across GPUs, with an all-reduce combining partial results. This parallelizes within a single layer's computation and is essential when a layer's weights alone exceed a single device's memory, but requires high-bandwidth, low-latency interconnect (e.g., NVLink within a server node) because it introduces communication inside every layer's forward and backward pass.

6.3 Pipeline parallelism

The model's layers are partitioned into sequential stages placed on different devices (or groups of devices), and micro-batches are streamed through the pipeline so multiple stages are active concurrently. This reduces per-device memory (only a subset of layers resides on each device) but introduces "pipeline bubbles" — idle time while the pipeline fills and drains — mitigated by interleaved scheduling strategies (e.g., 1F1B — one-forward-one-backward — scheduling) that keep more of the pipeline busy at the cost of implementation complexity.

6.4 Sequence/context parallelism

For very long context windows, even a single sequence's activations (particularly the attention matrix) can exceed device memory; sequence parallelism shards the sequence dimension itself across devices, with specialized communication patterns (e.g., ring-attention style exchanges of KV blocks) to compute exact attention over the full sequence collectively.

6.5 ZeRO / Fully Sharded Data Parallel (FSDP)

ZeRO (Zero Redundancy Optimizer) and FSDP eliminate data parallelism's redundant memory by sharding optimizer states, gradients, and optionally parameters themselves across data-parallel ranks, reconstructing the needed full tensors on-the-fly via all-gather just before they're used and freeing them immediately after:

In practice, frontier training runs combine several of these strategies simultaneously — commonly referred to as "3D" or "4D parallelism" (data + tensor + pipeline + sequence/expert parallelism) — with the specific combination tuned to the cluster's interconnect topology (favoring tensor parallelism within a fast-interconnect node, and data/pipeline parallelism across slower inter-node links).

6.6 Checkpointing and fault tolerance

Multi-week jobs across thousands of accelerators experience hardware failures (GPU ECC errors, node network drops, storage hiccups) as a statistical certainty rather than an edge case. Mitigations include: frequent, asynchronous checkpointing of model and optimizer state to persistent storage so a failure loses at most a bounded number of steps; automated health-checking and straggler/faulty-node detection that can evict and replace a bad node mid-job; and elastic or automatically-resuming training orchestration so a job restarts from the last good checkpoint without manual intervention. A single corrupted node silently producing wrong gradients (rather than crashing outright) is a particularly dangerous failure mode, since it can degrade the run without an obvious error signal — motivating numerical health checks (NaN/Inf detection, gradient-norm anomaly detection) at every step.

6.7 Activation memory management

Beyond parameters and optimizer state, intermediate activations needed for the backward pass consume substantial memory, especially at long context lengths. Activation checkpointing (gradient checkpointing) discards most intermediate activations during the forward pass and recomputes them during backward, trading extra compute (roughly one additional forward pass) for a large memory reduction, and is standard practice for large-context, large-model training.

7. Scaling Laws and Compute-Optimal Training

Empirical scaling laws describe how pretraining loss falls as a smooth, predictable power-law function of model size N (parameters), dataset size D (tokens), and compute C, over many orders of magnitude — enabling teams to extrapolate from small, cheap experiments to plan large, expensive runs before committing the full budget.

7.1 The Kaplan et al. scaling laws

Early large-scale scaling studies found that test loss follows separate power laws in N, D, and C when the other factors are not the bottleneck, of the rough form L(N) ≈ (Nc/N)α, and similarly for D and C, with exponents empirically around 0.05–0.095 depending on the factor and setup. A key early-era conclusion drawn from these laws was that, at a fixed compute budget, it was more efficient to train very large models on comparatively fewer tokens than to fully "use up" available data on a smaller model — which shaped a generation of models trained on relatively few tokens per parameter.

7.2 Chinchilla / compute-optimal scaling

Subsequent work (the "Chinchilla" study) revisited this with a more careful experimental protocol — sweeping model size and token count independently at many fixed compute budgets (an "IsoFLOP" analysis) rather than extrapolating a single fixed-token-count curve — and found that many earlier large models were significantly undertrained relative to their parameter count. The compute-optimal finding is that model size and training tokens should scale roughly in tandem: doubling the compute budget should roughly double both parameters and training tokens, rather than overwhelmingly favoring parameters. This produced the widely cited compute-optimal rule of thumb of training on roughly 20 tokens per parameter for compute-optimal loss at a given budget — e.g., a 70-billion-parameter compute-optimal model would be trained on roughly 1.4 trillion tokens.

7.3 Beyond "compute-optimal": inference-aware training

Compute-optimal scaling minimizes training loss for a fixed training compute budget, but it does not account for the far larger cumulative cost of serving a model at inference across millions of users over its deployment lifetime. Because a smaller model trained on many more tokens than the "compute-optimal" point continues to improve in quality — just at a diminishing, sub-compute-optimal rate of return per additional training FLOP — and because a smaller model is cheaper per token at inference, many production model families deliberately train well past the Chinchilla-optimal token count ("overtraining" relative to pure training-compute efficiency) to minimize total lifecycle cost. This is why publicly released open models are often trained on several trillion to tens of trillions of tokens even at parameter counts where the classic compute-optimal count would suggest far fewer.

7.4 Data-constrained scaling

When abundant unique high-quality data is not available, "data-constrained" scaling studies analyze the returns to repeating (multi-epoching) a fixed corpus versus training a smaller model, finding that a modest number of repeated epochs (roughly up to four, in the studies commonly cited) captures most of the benefit of fresh unique data, with returns degrading substantially at larger repeat counts — an important consideration as frontier labs approach the limits of readily available high-quality natural text and increasingly supplement with synthetic and multimodal data.

7.5 Using scaling laws in practice

Teams fit scaling curves from a grid of smaller training runs (varying N and D at fixed architecture family and fixed hyperparameter-scaling rules) and extrapolate the fitted power law to predict the loss, and often the downstream benchmark performance, of a target large run before committing the budget — used both to choose the model-size/token-count point for a given compute budget and to sanity-check that a large run's observed loss curve is tracking the predicted trajectory (a divergence from the predicted curve is itself a diagnostic signal of a data or implementation problem).

8. Stability and Failure Modes

8.1 Loss spikes and divergence

Sudden, large increases in training loss ("spikes") occur more frequently as model scale and depth increase, and if uncorrected can permanently destabilize training (the model may not recover, or may recover but at a permanently degraded loss trajectory). Root causes include: an anomalous data batch (e.g., an unusually repetitive or degenerate document), interaction between the Adam optimizer's second-moment estimate and a sudden large gradient, and numerical precision issues in low-precision training. Mitigations include gradient clipping, z-loss regularization, more conservative learning rates or additional warmup, and — for the most severe/reproducible spikes — simply skipping or down-weighting the offending batch and resuming from the last checkpoint before the spike.

8.2 Precision-related failures

FP16's narrow exponent range makes it prone to gradient underflow (very small gradients flush to zero, silently halting learning for affected parameters) without properly tuned dynamic loss scaling. Even in BF16, the reduced mantissa can cause updates smaller than the representable precision step to be silently dropped when accumulated directly in BF16, which is why master weights and optimizer accumulation are usually kept at higher precision.

8.3 Data-induced failure modes

Leaked evaluation data inflates benchmark scores in a way that is easy to miss until an unusually strong result on a specific benchmark, relative to correlated benchmarks, prompts an audit. Duplicated content trains the model to over-memorize specific spans, both wasting compute and increasing verbatim-regurgitation risk. Corrupted or mis-encoded text (e.g., garbled Unicode from a bad extraction step) that slips past filtering can silently degrade fluency in affected languages or scripts.

8.4 Mode collapse and repetition

Distinct from training loss instability, degenerate repetition (the model getting stuck generating the same token or phrase in a loop) is primarily a sampling/decoding phenomenon at inference time rather than a training failure per se, though it is exacerbated by training issues such as insufficiently diverse data or excessive exposure to templated/boilerplate text that was not adequately filtered.

8.5 Monitoring

Production pretraining runs continuously track: training and held-out validation loss curves (compared against the scaling-law-predicted trajectory), gradient norm (pre- and post-clip), parameter and activation norms per layer (to catch a specific layer diverging before it manifests as an overall loss spike), learning-rate and optimizer-state statistics, hardware health metrics (GPU utilization, ECC error counts, network throughput), and periodic downstream-benchmark evaluation on held-out checkpoints to catch quality regressions that pure loss tracking would miss (loss and downstream task performance can decouple, especially as models grow).

9. Evaluating a Pretrained Base Model

9.1 Intrinsic evaluation

Held-out perplexity/cross-entropy loss on a representative, decontaminated validation split is the most direct measure of how well the model has fit the target distribution, and is the metric scaling laws are typically fit against. It is comparable across checkpoints of the same tokenizer and data distribution but not meaningfully comparable across models using different tokenizers or evaluation corpora.

9.2 Downstream/extrinsic evaluation

Because a base model has not been instruction-tuned, it is typically evaluated via few-shot prompting (a handful of input-output examples prepended to the prompt) rather than zero-shot instructions, on benchmark suites spanning knowledge (closed-book QA), reasoning (multi-step arithmetic and logic benchmarks), reading comprehension, and code generation. Results are sensitive to prompt formatting, number of few-shot examples, and answer-extraction methodology, which is why cross-paper benchmark comparisons must be treated cautiously unless the evaluation harness is identical.

9.3 Contamination auditing

Beyond the pretraining-time decontamination pass described earlier, released benchmark scores are typically accompanied (in rigorous releases) by an explicit contamination analysis quantifying n-gram overlap between the training corpus and each benchmark, since undetected contamination is one of the most common causes of inflated, non-reproducible reported capability.

9.4 Behavioral and safety-relevant probing

Even prior to any alignment post-training, base-model checkpoints are commonly probed for factual reliability patterns (confident hallucination on obscure entities), memorization/regurgitation of training text (via targeted extraction attacks that check whether specific known training documents can be reproduced verbatim), and bias patterns across demographic and topical axes — all of which inform both data-pipeline iteration for the next training run and the design of the post-training stage that follows.

10. From Base Model to Product

The pretrained checkpoint is a raw text-completion engine: prompted with a question, it is as likely to continue with a related but unhelpful continuation (e.g., a list of similar questions, as would appear on a web page) as with a direct answer. Converting it into a deployable assistant requires post-training, most commonly a pipeline of: supervised fine-tuning (SFT) on curated instruction-response pairs to establish the assistant "shape" of interaction; preference-based alignment (RLHF, or direct preference optimization variants) to shape style, helpfulness, and refusal behavior according to human or model-based preference judgments; and often further specialized stages (tool-use training, long-context fine-tuning, safety-specific fine-tuning) layered on top. Because pretraining sets the ceiling on latent capability, weaknesses introduced at the pretraining stage — gaps in domain coverage, tokenizer inefficiencies for a target language, insufficient code representation, contamination-inflated apparent capability — cannot be fully corrected by post-training, which motivates the very large engineering investment described in the sections above going into the pretraining stage itself.