Pretraining

Objective and Outcome

Pretraining teaches a model general language understanding by predicting the next token in a sequence. Given tokens t₁...t_n, the model learns P(t_n+1 | t₁...t_n). Minimizing cross-entropy over trillions of tokens produces a base model that encodes grammar, facts, reasoning patterns, and world knowledge implicitly in its weights.

The base model is not yet a helpful assistant. It completes text rather than following instructions. Downstream fine-tuning and alignment reshape behavior, but pretraining sets the ceiling for knowledge breadth and reasoning capacity.

Data Pipeline

Pretraining data is the dominant quality lever. Typical sources include web crawls, books, academic papers, code repositories, and licensed datasets. The pipeline usually involves:

• Crawling and ingestion at web scale
• Deduplication (exact and fuzzy) to avoid memorization and wasted compute
• Quality filtering via heuristics, classifiers, and perplexity thresholds
• PII and toxic content removal for safety and compliance
• Domain mixing with tuned sampling ratios (e.g., more code for coding models)

Refer: The Pile dataset paper

Training Configuration

Hyperparameters that matter

• Learning rate schedule - warmup then cosine or linear decay
• Batch size - often maximized within memory, sometimes with gradient accumulation
• Sequence length - context window baked in at pretrain time (or extended later)
• Precision - mixed FP16/BF16 training with loss scaling where needed
• Tokenizer - fixed before training; changing it requires retraining embeddings

Training runs for a defined number of tokens seen per parameter (Chinchilla-optimal ratios are a common reference point). Checkpointing, validation loss tracking, and divergence detection are critical on multi-week runs.

# Hugging Face Trainer — typical pretraining knobs
training_args = TrainingArguments(
    output_dir="./checkpoints",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=3e-4,
    warmup_ratio=0.01,
    lr_scheduler_type="cosine",
    bf16=True,
    max_steps=100_000,
    save_steps=1000,
    logging_steps=10,
)

Scaling Laws and Compute Planning

Loss improves as a power law with model size, data, and compute until hitting diminishing returns. Planning a pretrain run means estimating FLOPs, GPU count, network bandwidth, storage for checkpoints, and failure recovery time. A single node failure on a thousand-GPU job must not corrupt the entire run.

Most product teams do not pretrain from scratch. They fine-tune open base models. Pretraining knowledge still matters for evaluating base checkpoints, understanding data contamination in benchmarks, and deciding when custom pretraining is worth the cost.

Stability and Failure Modes

Large training runs can diverge (loss spikes), underflow in low precision, or develop repetitive generation (mode collapse in sampling). Mitigations include gradient clipping, careful initialization, z-loss regularization, and monitoring activation norms per layer. Data bugs (e.g., leaked eval sets, duplicated epochs) show up as suspiciously good benchmark scores later.

From Base Model to Product

After pretraining, the checkpoint is evaluated on perplexity and downstream benchmarks, then passed to supervised fine-tuning for instruction following and post-training for alignment. Understanding pretraining helps you interpret why a base model hallucinates confidently, why code performance correlates with code token fraction, and why extending context may require continued pretraining or specialized long-context fine-tuning.