What Makes a Model "Large"
Large language models are transformer-based neural networks trained on massive text corpora to predict the next token. "Large" refers to both parameter count (billions to hundreds of billions) and training compute (FLOPs, GPU-hours). Scale unlocks emergent behaviors: in-context learning, chain-of-thought reasoning, tool use, and broad task generalization without task-specific training.
Capability is not only about parameters. Data quality, training duration, architecture choices (context length, MoE vs dense), and post-training alignment all shift the effective intelligence of a model at a given size.
Refer: Chinchilla scaling laws
Scaling Laws
Empirical scaling laws show that loss decreases predictably as you increase model size, dataset size, and compute budget. The Chinchilla paper argued many models were under-trained: for a fixed compute budget, smaller models trained on more tokens often outperform larger models trained on fewer tokens.
# Chinchilla rule of thumb: train on ~20 tokens per parameter
# Training FLOPs ≈ 6 × N × D (N = params, D = training tokens)
params = 7_000_000_000
tokens = 20 * params # ~140B tokens for a 7B model
Practical implications for engineering teams:
- Bigger is not always better per dollar at inference time
- Training token count should scale with parameter count
- Evaluation on target tasks matters more than parameter bragging rights
Model Lifecycle
A production-grade LLM typically moves through these stages:
- Pretraining on broad internet, books, and code corpora
- Supervised fine-tuning (SFT) on instruction-response pairs
- Post-training alignment via RLHF, DPO, or similar preference optimization
- Evaluation on benchmarks and custom task suites
- Deployment behind APIs or on owned infrastructure
- Monitoring and iteration as usage patterns and failure modes surface
Open Weights vs API Models
API-hosted models
Providers like OpenAI, Anthropic, and Google host models behind HTTP APIs. You get fast iteration, no GPU ops burden, and regular capability upgrades. Tradeoffs include data residency policies, per-token cost at scale, rate limits, and less control over weights or fine-tuning depth.
Open-weight models
Models such as LLaMA, Mistral, and Qwen ship downloadable weights. You control deployment, fine-tuning, and data flow. Tradeoffs include infrastructure cost, safety tuning responsibility, and keeping up with upstream releases.
Many products use a hybrid: API models for complex queries, open models for high-volume or privacy-sensitive paths.
Context Windows and Modalities
Context window is the maximum tokens a model can attend to in one forward pass. Long-context models (32K, 128K, 1M+) enable document QA, large codebases, and multi-turn agents without aggressive summarization. Long context increases KV cache memory and latency linearly (or worse without optimized attention).
Multimodal LLMs extend the same autoregressive core to images, audio, or video by encoding non-text inputs into token-like embeddings fused with text tokens. The engineering surface expands to preprocessing pipelines, modality-specific encoders, and combined safety filters.
Choosing a Model for Your Product
Match model to workload: latency-sensitive chat may need a smaller instruct model; offline batch summarization can use a larger model; code tasks benefit from code-heavy training data; regulated industries may require on-prem open weights. Evaluate on your own data early. Public leaderboard scores are a starting point, not a substitute for domain-specific eval sets.