Large Language Models
Architecture, scale, and capability questions specific to large generative models.
What technically distinguishes an LLM from a smaller language model?
There is no single hard cutoff, but LLMs typically have billions of parameters, are trained on trillions of tokens, and exhibit strong few-shot and instruction-following behavior that smaller models lack. Scale unlocks emergent patterns: better in-context learning, broader world knowledge stored in weights, and more robust reasoning when paired with the right post-training.
What are scaling laws and why do labs obsess over them?
Scaling laws describe predictable loss improvements as you increase model size (N), dataset size (D), and compute (C). They let teams forecast whether a larger training run is worth the budget before spending millions on GPUs. Chinchilla showed many models were under-trained for their size; modern labs often train smaller models longer to optimize inference cost.
What is the difference between total parameters and active parameters (especially in MoE)?
Total parameters count every weight in the model. Active parameters are only those used in a single forward pass. Mixture-of-Experts models may have 100B+ total parameters but activate only a subset (e.g. 10-15B) per token via a router, giving large-model capacity without proportional inference FLOPs.
How does context window length affect LLM system design?
Longer context lets you pack more retrieved documents, conversation history, and tool outputs into one prompt, but increases prefill compute and KV cache memory quadratically or linearly depending on optimizations. Engineering choices include: chunking/RAG vs long context, prompt compression, KV cache quantization, and models with native long-context support (RoPE scaling, YaRN, etc.).
What are emergent abilities and should you rely on them in production?
Emergent abilities are capabilities (e.g. multi-step arithmetic, instruction following) that appear suddenly at certain model scales on benchmarks. They are useful for model selection but risky as a production strategy: behavior can be inconsistent, hard to eval, and may not transfer across model versions. Prefer explicit fine-tuning, RAG, or tooling over hoping emergence.
Decoder-only vs encoder-decoder LLMs: which dominates and why?
Decoder-only (GPT, LLaMA, Claude-style) dominates generative AI because one architecture handles pretraining, fine-tuning, and inference uniformly, scales well, and supports simple autoregressive APIs. Encoder-decoder still wins some seq2seq tasks (translation, summarization with long inputs) but most new frontier LLMs are decoder-only.
What is a foundation model vs a chat model vs an instruct model?
A foundation/base model is raw pretrained weights (next-token prediction). An instruct/SFT model is fine-tuned on instruction-response pairs. A chat/aligned model adds RLHF, DPO, or similar preference optimization for helpful, safe dialogue. Production assistants almost always use chat models; base models are for further fine-tuning or research.
Open-weight vs closed API models: engineering implications?
Open weights (LLaMA, Mistral, Qwen) give full control: self-hosting, fine-tuning, no data leaving your infra, predictable unit economics at scale. Closed APIs (GPT-4, Claude) offer frontier quality without GPU ops but add vendor lock-in, rate limits, and per-token cost. Many production systems use a router: closed API for hard tasks, open model for high-volume easy tasks.
What is model collapse and when does it matter?
Model collapse occurs when models are trained recursively on AI-generated text, causing diversity and quality to degrade over generations. It matters for synthetic data pipelines and web-scale scraping as AI content grows. Mitigations: filter synthetic data, mix real human data, and monitor distributional drift in training corpora.