Large Language Models: A Deep Dive

A researcher-level walkthrough of transformers, attention, scaling laws, training pipelines, and inference — with the math made explicit, and a plain-English recap at the end.

Contents

1. What a Language Model Actually Computes

At its core, a large language model (LLM) is a parametric estimate of a conditional probability distribution:

P(x_t | x_1, x_2, ..., x_{t-1}; θ)

Given a sequence of tokens x_1 ... x_{t-1}, the model outputs a probability distribution over the next token x_t, parameterized by weights θ (often tens to hundreds of billions of numbers). Everything else — chat behavior, code generation, reasoning, tool use — is a consequence of this single objective applied at massive scale, not a separately engineered capability.

The joint probability of an entire sequence factorizes autoregressively via the chain rule of probability:

P(x_1, ..., x_T) = Π_{t=1}^{T} P(x_t | x_1, ..., x_{t-1})

This factorization is why LLMs generate text one token at a time, left to right — it is baked into the mathematical decomposition of the objective, not an arbitrary implementation choice.

The architecture that computes P(x_t | x_<t; θ) at scale, efficiently and trainably, is the Transformer, introduced in Vaswani et al., 2017 ("Attention Is All You Need"). Almost every LLM in production today — GPT, Claude, Gemini, LLaMA, Mistral — is a descendant of this architecture, typically the decoder-only variant.

Refer: Attention Is All You Need (Vaswani et al., 2017)

2. Tokenization: Turning Text Into Numbers

Neural networks operate on continuous vectors, not characters. The first step is converting raw text into a sequence of integer IDs from a fixed vocabulary. Modern LLMs use subword tokenization — most commonly Byte-Pair Encoding (BPE) or SentencePiece/Unigram models.

Why not whole words? Vocabulary would explode (millions of word forms across languages, typos, rare terms), and out-of-vocabulary words would be unrepresentable.

Why not individual characters? Sequences would become extremely long, straining compute (attention cost is quadratic in sequence length — see Section 18), and it would be harder for the model to learn semantic units.

BPE algorithm sketch:

The result: common words become single tokens ("the", "is"), while rare words are split into meaningful fragments ("tokenization" → "token" + "ization"). This gives a good trade-off between sequence length and vocabulary size, and guarantees any input string is representable (fallback to raw bytes).

Each token ID is just an index — the meaning is not yet encoded anywhere. That's the job of the embedding layer.

3. Embeddings and the Representation Space

Each token ID i is mapped to a dense vector via an embedding matrix E ∈ R^{V × d}, where V is vocabulary size and d is the model's hidden dimension (e.g., 4096, 8192, 12288 depending on scale).

e_i = E[i, :]   ∈ R^d

This embedding is a learned lookup table — nothing more than a matrix multiplication with a one-hot vector, in effect. Initially these vectors are random; training shapes the space so that tokens used in similar contexts end up geometrically close (the distributional hypothesis: "a word is characterized by the company it keeps," Firth, 1957).

Critically, the raw embedding carries no information about position. e_i for the token "bank" is identical whether it's the first word in the sequence or the fiftieth. Position must be injected separately (Section 7), because the core attention operation itself is permutation-invariant — it treats its input as a set, not a sequence, unless told otherwise.

4. The Core Problem Attention Solves

Before transformers, sequence models were dominated by RNNs/LSTMs, which process tokens one at a time, carrying a hidden state forward. Two structural problems:

Attention was originally introduced (Bahdanau et al., 2014) as a patch on top of RNNs, letting a decoder "look back" at all encoder hidden states weighted by relevance rather than relying solely on a single compressed context vector. Vaswani et al.'s insight, formalized in the 2017 paper, was radical in its simplicity: throw away the recurrence entirely and build the whole model out of attention. Hence the title.

Core idea: for every token, directly compute a weighted combination of every other token's representation, where the weights are learned functions of content similarity — not fixed by distance, not degraded by sequential propagation. This is what enables full parallelization during training and near-arbitrary long-range dependency modeling in a single layer.

5. Scaled Dot-Product Attention — Full Derivation

This is the mathematical heart of the transformer. Every token's representation is projected into three different vectors using three separate learned weight matrices:

Q = X W_Q     (Query)
K = X W_K     (Key)
V = X W_V     (Value)

Where X ∈ R^{n × d} is the matrix of input embeddings for a sequence of length n, and W_Q, W_K, W_V ∈ R^{d × d_k} are learned projection matrices.

Intuition first. Think of it as a soft, differentiable database lookup:

For every pair of tokens (i, j), we measure how relevant token j is to token i by taking the dot product of i's query and j's key: q_i · k_j. A large dot product means the vectors are well-aligned in the learned space — high relevance.

Step 1 — Similarity scores. Compute all pairwise dot products at once via matrix multiplication:

S = Q K^T     ∈ R^{n × n}

S[i, j] is the raw (unnormalized) relevance of token j to token i.

Step 2 — Scaling. Divide by √d_k:

S_scaled = S / √d_k

Why this specific scaling? If q and k are vectors whose components are independent random variables with mean 0 and variance 1, their dot product q · k = Σ_{l=1}^{d_k} q_l k_l has variance d_k (variance of a sum of d_k independent products, each with variance 1). As d_k grows, the dot products grow large in magnitude. Large logits pushed into a softmax produce an extremely peaked, near one-hot distribution, which drives the gradient of the softmax toward zero almost everywhere (saturated softmax → vanishing gradients). Dividing by √d_k renormalizes the variance of the dot product back to approximately 1, regardless of dimensionality, keeping the softmax in a well-conditioned regime throughout training. This single line is why deep transformers with high-dimensional heads remain trainable at all.

Step 3 — Softmax normalization. Convert scores into a valid probability distribution over "which tokens to attend to," row-wise:

A[i, j] = softmax_j(S_scaled[i, :])
        = exp(S_scaled[i,j]) / Σ_k exp(S_scaled[i,k])

Each row of A now sums to 1 — token i distributes 100% of its "attention budget" across all tokens in the sequence, proportional to relevance.

Step 4 — Weighted aggregation of values. Use these attention weights to combine the Value vectors:

Attention(Q, K, V) = A · V = softmax(QK^T / √d_k) · V

The output for token i is a convex combination of every token's Value vector, weighted by learned relevance. This entire operation — from X to output — is fully differentiable and computed with two matrix multiplications and one softmax, meaning it is embarrassingly parallel across the sequence dimension (unlike an RNN) and maps extremely efficiently onto GPU matrix-multiply hardware.

Computational cost. QK^T is an n × n matrix, so both compute and memory scale as O(n² d). This quadratic term in sequence length n is the central scaling bottleneck of transformers, discussed at length in Section 18.

6. Multi-Head Attention

A single attention operation forces all relational information (syntax, coreference, semantic similarity, positional proximity, etc.) through one shared subspace. Multi-head attention instead runs several smaller attention operations in parallel, each in its own learned subspace, then combines them.

head_i = Attention(X W_Q^i, X W_K^i, X W_V^i),   i = 1 ... h
MultiHead(X) = Concat(head_1, ..., head_h) W_O

If the model dimension is d and there are h heads, each head typically operates in dimension d_k = d / h, so the total compute is comparable to a single full-dimension attention operation, but the representational capacity is richer: empirically, different heads specialize — some attend to adjacent tokens (local syntax), others to the previous occurrence of the same word (coreference-like behavior), others to sentence-initial tokens (a global "attention sink"), and so on. This specialization was demonstrated via attention-pattern visualization in the original paper and extensively in later interpretability work.

W_O ∈ R^{(h·d_k) × d} is a final learned linear projection that mixes information across heads back into the shared residual stream (see Section 9).

7. Positional Information: Sinusoidal, Learned, RoPE, ALiBi

Because attention is permutation-invariant (shuffle the input tokens and, absent positional signal, you get the same set of pairwise interactions), the model has no innate sense of order. Position must be injected explicitly. There have been several major approaches:

Sinusoidal positional encoding (original paper). Add a fixed (non-learned), deterministic vector to each token embedding based on its position pos:

PE(pos, 2i)   = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

This uses a different sinusoid frequency per dimension pair. The authors chose this because for any fixed offset k, PE(pos+k) can be expressed as a linear function of PE(pos) (a rotation), which they hypothesized would let the model easily learn to attend by relative position. It also generalizes to sequence lengths not seen during training, at least in principle.

Learned absolute positional embeddings. Simply learn an embedding table indexed by position, exactly like the token embedding table. Simpler, but does not generalize beyond the trained maximum length.

Rotary Positional Embedding (RoPE) — used in most modern LLMs (LLaMA, GPT-NeoX-style models, and others). Instead of adding a positional vector, RoPE rotates the Query and Key vectors in 2D subspaces by an angle proportional to their absolute position, such that the dot product q_i · k_j after rotation depends only on the relative position (i - j), not on absolute positions:

q'_m = R(mθ) q_m
k'_n = R(nθ) k_n
q'_m · k'_n  =  f(q_m, k_n, m - n)

where R(·) is a block-diagonal rotation matrix. This relative-position property tends to generalize better to longer sequences than absolute schemes and is a major reason RoPE became the default choice.

ALiBi (Attention with Linear Biases). Skips positional embeddings on the input entirely and instead directly biases the attention scores by a linear penalty proportional to distance:

score(i,j) = q_i · k_j / √d_k − m · |i − j|

where m is a fixed, head-specific slope. This is cheap, requires no extra parameters, and extrapolates gracefully to sequences longer than seen during training, since it just makes distant tokens progressively less attractive rather than encoding position abstractly.

8. The Position-Wise Feed-Forward Network

Every transformer block also contains a feed-forward network (FFN), applied independently and identically to every token position (hence "position-wise"):

FFN(x) = W_2 · activation(W_1 x + b_1) + b_2

W_1 ∈ R^{d × d_ff} expands the dimension (typically d_ff = 4d), an activation function is applied elementwise, and W_2 ∈ R^{d_ff × d} projects back down. The original paper used ReLU; modern LLMs typically use GeLU or SwiGLU variants, which are smoother and empirically improve optimization and downstream quality:

SwiGLU(x) = (Swish(x W_a)) ⊗ (x W_b)
Swish(z) = z · sigmoid(z)

Conceptually: attention moves information between token positions; the FFN processes information within a token position. They are complementary operations, and interleaving them repeatedly across many layers is what allows the network to build increasingly abstract representations — early layers tend to capture local syntax and surface patterns, deeper layers capture longer-range semantics, discourse structure, and task-relevant abstractions.

The FFN typically accounts for roughly two-thirds of a transformer's parameters, since d_ff is usually 4x d and there are two such matrices per layer.

9. Residual Connections and Layer Normalization

Stacking dozens of attention + FFN blocks (modern LLMs commonly have 30–120+ layers) creates a very deep network, which is notoriously hard to train due to vanishing/exploding gradients. Two techniques make this tractable:

Residual (skip) connections. Instead of a layer computing y = F(x), it computes y = x + F(x). During backpropagation, the gradient has a direct additive path back to earlier layers (∂y/∂x includes an identity term), so gradients don't have to survive being multiplied through dozens of nonlinear transformations to reach early layers. This is the single most important trick that makes 100+ layer networks trainable at all (originally from ResNets, He et al., 2015).

Layer Normalization. Normalizes activations across the feature dimension for each token independently:

LN(x) = γ · (x − μ) / √(σ² + ε) + β

where μ, σ² are the mean and variance computed across the d feature dimensions of a single token's vector, and γ, β are learned scale/shift parameters. This stabilizes the distribution of activations flowing into each sublayer, which in turn stabilizes gradients.

Post-LN vs Pre-LN. The original paper applied LayerNorm after the residual addition ("Post-LN"):

x_{l+1} = LN(x_l + Sublayer(x_l))

Most modern large-scale LLMs instead use Pre-LN, normalizing before the sublayer:

x_{l+1} = x_l + Sublayer(LN(x_l))

Pre-LN keeps the residual stream itself un-normalized and additive end-to-end, which empirically gives much more stable training at large depth and removes the need for careful learning-rate warmup tuning that Post-LN required. Many current models (LLaMA-family and others) further replace LayerNorm with RMSNorm, which normalizes only by root-mean-square magnitude (dropping the mean-centering step), which is cheaper and empirically just as effective:

RMSNorm(x) = γ · x / √(mean(x²) + ε)

10. The Full "Attention Is All You Need" Architecture

The original transformer is an encoder-decoder model, designed for sequence-to-sequence tasks like machine translation. It's worth dissecting fully, since almost every architectural component used in modern LLMs traces back to it.

INPUT SEQUENCE                         OUTPUT SEQUENCE (shifted right)
     │                                          │
Token Embed + PosEnc                    Token Embed + PosEnc
     │                                          │
┌────■───────┒                      ┌─────────■────┒
│   ENCODER     │                      │      DECODER      │
│  (× N layers) │                      │    (× N layers)   │
│               │                      │                    │
│ Multi-Head    │                      │ Masked Multi-Head  │
│ Self-Attention│                      │ Self-Attention     │
│      │        │                      │       │            │
│  Add & Norm   │                      │   Add & Norm       │
│      │        │                      │       │            │
│      │        │        K,V from      │ Cross-Attention    │
│      │        │◄───────encoder────│ (Q from decoder,   │
│      │        │        output        │  K,V from encoder) │
│      │        │                      │       │            │
│      │        │                      │   Add & Norm       │
│      │        │                      │       │            │
│  Feed-Forward │                      │  Feed-Forward       │
│      │        │                      │       │            │
│  Add & Norm   │                      │   Add & Norm        │
└────╨───────┘                      └─────────╨─────────┘
     │                                            │
     └──────╨ encoder output (context) ──────╨
                                                    │
                                          Linear → Softmax
                                                    │
                                          Output probabilities
                                          over next token

Encoder stack. Each of the N identical layers has two sublayers: (1) multi-head self-attention, where queries, keys, and values all come from the same input sequence — every input token can attend to every other input token, in both directions (no masking), and (2) the position-wise FFN. Each sublayer is wrapped in a residual connection + LayerNorm. The encoder's job is to build a rich, bidirectional contextual representation of the entire input sequence.

Decoder stack. Each layer has three sublayers: (1) masked multi-head self-attention over the output sequence generated so far, (2) multi-head cross-attention, where the queries come from the decoder but the keys and values come from the encoder's output — this is how the decoder "reads" the source sequence while generating the target, and (3) the position-wise FFN.

Masking in the decoder. The self-attention in the decoder must not be allowed to look at future tokens (that would be cheating — trivially copying the answer during training). This is enforced with a causal mask: before the softmax, all positions j > i are set to −∞ in the score matrix S, so exp(−∞) = 0 and those tokens receive exactly zero attention weight. This is one of the most important structural choices in the whole architecture, and it's the piece that carries forward almost unchanged into modern decoder-only LLMs.

Final projection. The decoder's final hidden states are projected through a linear layer back to vocabulary size V, then softmaxed to produce P(x_t | x_<t). In many implementations this final projection matrix is tied (weight-shared) with the input embedding matrix E, which reduces parameter count and has been shown to improve quality.

11. From Encoder-Decoder to Decoder-Only: How GPT-Style LLMs Diverge

Machine translation needs an encoder (to fully understand a fixed source sentence bidirectionally) and a decoder (to generate a target sentence conditioned on it). But most of what we now call "LLMs" — GPT, Claude, LLaMA, Gemini — are decoder-only. Why?

The generative task these models are trained on is open-ended text continuation, where there's no clean separation between "source" and "target": the entire prompt and the entire response are just one long token sequence to be modeled autoregressively. There's no need for a separate bidirectional encoder module — the causal (masked) self-attention decoder alone, applied to the whole sequence (prompt + generation), is sufficient, and it's architecturally simpler:

[prompt tokens] [response tokens]
        └── all attend causally, left to right, over the whole thing ──╨

A decoder-only transformer layer is therefore just: masked multi-head self-attention → Add & Norm → FFN → Add & Norm, repeated N times, with no cross-attention sublayer and no separate encoder. This simplification, combined with removing the encoder-decoder split, is essentially what "GPT" (Radford et al., 2018) did to the original transformer — reusing the decoder stack from Vaswani et al. almost verbatim, minus cross-attention, and training it purely as a next-token predictor over huge unlabeled text corpora. Every "chat" LLM you interact with today is architecturally this decoder-only stack, scaled up by orders of magnitude in parameters and training data, with additional alignment stages layered on top (Section 15).

Other architectural families exist for completeness: encoder-only models (like BERT) drop the causal mask and are trained with masked language modeling (predict randomly hidden tokens using full bidirectional context) — excellent for embeddings and classification, but not naturally suited to open-ended generation since they don't factorize autoregressively.

12. The Training Objective: Cross-Entropy and Teacher Forcing

Given the autoregressive factorization from Section 1, training minimizes the negative log-likelihood of the training corpus:

L(θ) = − (1/T) Σ_{t=1}^{T} log P(x_t | x_1, ..., x_{t-1}; θ)

This is exactly the cross-entropy loss between the model's predicted distribution and the one-hot "true" next token, averaged over every position in every sequence in the training batch. Minimizing this is equivalent to maximizing the likelihood of the training data under the model, and it's also directly related to perplexity, a common evaluation metric:

Perplexity = exp(L)

Lower perplexity means the model is, on average, less "surprised" by the actual next token — a perplexity of 1 would mean perfect prediction; a perplexity of V (vocab size) is equivalent to random guessing.

Teacher forcing. During training, the model is always fed the ground-truth previous tokens as input when predicting position t, never its own (possibly wrong) previous predictions. Combined with the causal mask, this means the entire loss for an n-token sequence can be computed in a single forward pass — every position's prediction and loss term is computed in parallel, not sequentially, because the mask already guarantees position t cannot see positions > t. This is a crucial efficiency win: training a decoder-only transformer is fully parallel across both the batch dimension and the sequence dimension, which is precisely what makes training on trillions of tokens on GPU/TPU clusters computationally feasible. (This is in contrast to inference, where generation genuinely must proceed token by token, since future tokens don't exist yet — Section 17.)

13. Optimization Mechanics: Adam, Warmup, and Stability

LLMs are trained with variants of Adam (Kingma & Ba, 2014), specifically AdamW (Adam with decoupled weight decay), which maintains per-parameter adaptive learning rates using running estimates of the first and second moments of the gradient:

m_t = β1 m_{t-1} + (1 − β1) g_t          (momentum)
v_t = β2 v_{t-1} + (1 − β2) g_t²         (adaptive scale)
θ_t = θ_{t-1} − η · m̂_t / (√v̂_t + ε) − η·λ·θ_{t-1}   (weight decay term)

Two stability techniques are essentially universal in LLM pretraining:

Learning-rate warmup then decay. The learning rate is linearly ramped up from ~0 over the first several hundred to few thousand steps, then decayed (cosine or linear schedule) over the rest of training. Early in training, gradients from randomly initialized weights can be large and noisy; warmup prevents an initial large update from destabilizing the whole optimization trajectory, especially important in Post-LN architectures and still commonly used even with the more stable Pre-LN.

Gradient clipping. Gradient norms are clipped to a maximum value (e.g., 1.0) before the optimizer step, to prevent rare large-magnitude gradient spikes — common in very deep, very large models — from causing loss divergence.

At the scale of frontier models, training runs span weeks to months across thousands of accelerators, and require additional systems-level techniques (mixed-precision arithmetic, activation checkpointing, model/data/pipeline parallelism, ZeRO-style optimizer state sharding) that are as much distributed-systems engineering as they are machine learning.

14. Scaling Laws: Kaplan and Chinchilla

A defining empirical discovery of the last several years is that transformer loss follows remarkably smooth power-law relationships with scale, allowing researchers to predict the performance of a not-yet-trained model.

Kaplan et al. (2020) found that loss L decreases as a power law in each of parameters N, dataset size D, and compute C, when the other two are not bottlenecking:

L(N) ≈ (N_c / N)^{α_N}
L(D) ≈ (D_c / D)^{α_D}

Their fitted exponents suggested that, for a fixed compute budget, it was more efficient to grow model size aggressively and train on relatively fewer tokens — this guided the design of very large models like GPT-3 (175B parameters, trained on roughly 300B tokens).

Chinchilla (Hoffmann et al., 2022) re-ran this analysis more carefully and reached a different conclusion: most large models of that era were significantly under-trained relative to their parameter count. For a fixed compute budget C ≈ 6ND (a standard approximation for transformer training FLOPs, where the factor of 6 accounts for the forward and backward pass matrix multiplications), the compute-optimal allocation is roughly:

N_opt ∝ C^{0.5}
D_opt ∝ C^{0.5}

i.e., parameters and training tokens should scale roughly equally, not skewed toward parameters. Their rule of thumb — train on roughly 20 tokens per parameter — reshaped how the field allocates compute. A 70B-parameter Chinchilla-optimal model trained on ~1.4 trillion tokens outperformed the much larger, less-tokens-per-parameter Gopher (280B) and GPT-3 (175B) on a broad evaluation suite, despite requiring far less inference compute.

A crucial nuance for practitioners: "compute-optimal" only minimizes training loss for a given training compute budget. It says nothing about inference cost, which is paid every single time the model is queried, potentially billions of times over its deployment life. This has pushed the field further: many current models are trained well beyond the naive Chinchilla-optimal token count ("overtrained" relative to N), deliberately trading extra (one-time) training compute for a smaller, cheaper-to-serve model that's just as capable — because amortized inference cost, not training cost, dominates total lifecycle cost for widely-deployed models.

15. The Modern Training Pipeline: Pretraining → SFT → RLHF/DPO

A raw pretrained model — trained purely on next-token prediction over broad internet-scale text — is a base model. It is extremely good at continuing text in the statistical pattern of its training distribution, but it has no notion of "being a helpful assistant." Ask it a question, and it may just as easily continue with a list of similar questions (because that pattern is common in its training data) rather than answering. Turning a base model into an assistant requires additional post-training stages:

1. Supervised Fine-Tuning (SFT). The base model is further trained on a smaller, curated dataset of (prompt, ideal response) pairs, using the exact same next-token cross-entropy loss as pretraining, but now the "ground truth" is a human- or model-written high-quality answer rather than arbitrary internet text. This teaches the model the format and register of instruction-following/dialogue.

2. Preference-based alignment: RLHF or DPO. SFT alone doesn't teach nuanced quality judgments (helpfulness, harmlessness, honesty, style preferences) — it only imitates fixed examples. Preference optimization instead trains on comparisons: given a prompt and two candidate responses, humans (or a trained reward model) indicate which is better.

3. Additional stages in practice commonly include: safety-specific fine-tuning/red-teaming rounds, tool-use/agentic training, rejection sampling against a reward model to generate higher-quality SFT data, and iterative rounds of feedback collection as the model itself is used to generate candidate responses for further human/AI review (a loop sometimes called "RLAIF" when the feedback signal comes from another model rather than a human).

16. Emergent Abilities and In-Context Learning

One of the most striking empirical phenomena in LLM research is in-context learning (ICL): a sufficiently large pretrained model can perform a new task purely from a handful of examples given in the prompt, with zero gradient updates to its weights.

Prompt:
"cheerful → happy
 furious → angry
 gigantic → "
Model completes: "huge"

No parameter update happened here — the model's forward pass alone, conditioned on the pattern present in the prompt, is enough to induce task-appropriate behavior. This wasn't explicitly engineered; it emerged from scale and training data diversity. Mechanistically, one influential hypothesis (Olsson et al., 2022, on "induction heads") is that certain attention head circuits learn during pretraining to implement a general pattern-completion algorithm — roughly, "find where this exact prefix occurred earlier in context, and copy what followed it" — and this generic copying/completion circuit, learned purely to minimize next-token loss on ordinary text, turns out to generalize into a surprisingly powerful few-shot learning mechanism.

Emergent abilities more broadly refers to capabilities (multi-step arithmetic, certain reasoning benchmarks) that appear to be near-zero for smaller models and then increase sharply once a model crosses a certain scale threshold — the capability appears to "emerge" rather than improve smoothly. This is an active and somewhat contested research area: some argue these are genuine phase transitions in capability; others (Schaefer et al., 2023) argue much of the apparent discontinuity is an artifact of using discontinuous evaluation metrics (like exact-match accuracy) on tasks where the underlying model quality is actually improving smoothly and continuously — a smooth increase in per-token probability of the correct answer can look like a sudden jump if you only measure whether the exact final answer is fully correct.

17. Inference: KV Caching, Sampling, and Decoding Strategies

Training computes losses for a whole sequence in parallel (Section 12), but inference is inherently sequential: to generate token t+1, you need to know token t, which the model just generated. This creates a major engineering challenge distinct from training.

KV Caching. Naively, generating each new token would require re-running the full forward pass over the entire sequence so far, recomputing Key and Value projections for every previous token — wasteful, since those don't change once computed (causal masking guarantees position i's Key/Value never depend on future tokens). Instead, implementations cache the Key and Value vectors for all previous tokens, and at each new step only compute Q, K, V for the single new token, then attend over the cached K/V plus the new one. This turns each generation step's attention cost from O(n²) (recomputing everything) down to O(n) (just the new token attending over cached history), at the cost of memory to store the growing KV cache — which itself becomes a major memory bottleneck at long context lengths and large batch sizes, motivating techniques like Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), where multiple query heads share a single set of Key/Value projections, shrinking the cache substantially with minimal quality loss.

Decoding strategies — how to turn the model's output probability distribution into an actual chosen next token:

Production chat systems typically combine temperature with top-p and/or top-k, tuned per use case (near-deterministic for code generation, more diverse for creative writing).

18. The Quadratic Bottleneck and Long-Context Solutions

As established in Section 5, standard self-attention costs O(n² · d) in compute and O(n²) in memory for the attention score matrix, where n is sequence length. Doubling context length quadruples the attention cost. This has driven substantial systems and algorithmic research:

FlashAttention (Dao et al., 2022) doesn't reduce the asymptotic O(n²) compute, but dramatically reduces the memory movement cost — it restructures the computation to avoid ever materializing the full n × n attention matrix in slow GPU high-bandwidth memory, instead computing attention in fused, tiled blocks that stay in fast on-chip SRAM, exploiting the fact that GPU kernels are frequently memory-bandwidth-bound rather than compute-bound. This alone gave several-fold real-world speedups and is near-universally used in production training and inference stacks today.

Sparse / local attention patterns restrict which token pairs are allowed to attend to each other at all (e.g., sliding-window local attention plus periodic global tokens, as in Longformer), trading some modeling flexibility for genuinely sub-quadratic asymptotic cost.

Linear attention / state-space models (e.g., Mamba, and other SSM-based architectures) reformulate the sequence-mixing operation entirely to avoid the pairwise n × n structure, achieving O(n) complexity, at the cost of a still-ongoing research question about whether they can match transformer quality at the largest scales, particularly for tasks requiring precise long-range retrieval.

Extending trained context length post-hoc: techniques like position-interpolation or NTK-aware scaling of RoPE's frequency base allow a model trained at one context length (say 4K tokens) to be adapted, often with modest additional fine-tuning, to operate at a much longer context length (say 128K), by rescaling the positional rotation frequencies rather than retraining from scratch.

19. Mixture-of-Experts and Modern Architectural Variants

Instead of every token passing through the same dense FFN, a Mixture-of-Experts (MoE) layer maintains multiple parallel FFN "experts," and a small learned router network selects (typically) the top 1 or 2 experts per token:

router_logits = x W_r
top_k experts selected via softmax(router_logits)
output = Σ_{i ∈ top_k} gate_i · Expert_i(x)

This decouples total parameter count from compute-per-token: a model can have, say, 8x the total parameters of a dense model, but since each token only activates 1-2 of the 8 experts, the FLOPs per token remain similar to a much smaller dense model. This gives more model capacity (useful for memorization/breadth of knowledge) without a proportional inference cost increase — though it introduces its own engineering challenges (load balancing across experts, routing collapse where the router degenerates to always picking the same expert, and increased memory footprint since all experts must be loaded even if not all are active per token).

Other widely-adopted architectural refinements beyond what's covered above include: Grouped-Query Attention (Section 17, memory-efficient KV caching), SwiGLU activations (Section 8), RMSNorm (Section 9), and various normalization-placement tweaks (e.g., normalizing Query/Key vectors themselves before the dot product, "QK-Norm," to further improve training stability at very large scale).

20. Multimodality

Extending an LLM beyond text follows the same underlying architecture, with the key change happening before the transformer stack: a modality-specific encoder converts non-text input (images, audio, video) into a sequence of continuous embedding vectors that live in (or are projected into) the same representational space as text token embeddings, and these are simply concatenated into the input sequence alongside text tokens.

For images, a common approach: split the image into fixed-size patches (e.g., 16×16 pixels), linearly project each flattened patch into a d-dimensional vector (this is the Vision Transformer / ViT patch-embedding approach, Dosovitskiy et al., 2020), optionally pass them through a dedicated vision encoder/adapter, and feed the resulting sequence of "image tokens" into the same transformer stack that processes text — the model then attends across text and image tokens uniformly, since after this initial encoding step they're just vectors in a shared embedding space to the rest of the network. Audio follows an analogous pattern (spectrogram patches or learned audio-token codecs). This is why multimodal capability, once the encoder/projection interface is built, requires no fundamental change to the transformer core itself — the attention and FFN mechanics described in Sections 5–9 are entirely modality-agnostic.

21. The Whole Story, End to End

Pull the threads together, from raw text to a generated reply:

A string of text is first tokenized into subword units and mapped to integer IDs. Each ID is looked up in an embedding table, producing a dense vector per token — but these vectors alone carry no sense of order, so a positional signal (sinusoidal, learned, or rotary) is woven in.

This sequence of vectors flows into a stack of dozens of identical transformer blocks. In each block, multi-head self-attention lets every token look at every earlier token (causally masked, so no peeking at the future), computing relevance via scaled dot products between Queries and Keys, and pulling in a weighted blend of Values — this is how "bank" figures out from context whether it means a riverbank or a financial institution, and how a pronoun ties back to its antecedent three sentences earlier. A residual connection carries the original signal forward unchanged alongside this new contextual information, and normalization keeps the numbers well-behaved. Then a feed-forward network processes each token's now-contextualized vector independently, transforming it through a learned nonlinear function — again wrapped in a residual connection and normalization. Stack this attention-then-FFN pattern thirty, sixty, a hundred-plus times, and the representations grow from raw token identity into something capturing grammar, facts, reasoning chains, and task intent.

At the very end, the final layer's output vector for the last token is projected back into vocabulary-sized logits and turned into a probability distribution via softmax — the model's best guess at what token comes next. During training, this whole pipeline runs in parallel across an entire sequence at once, and the difference between predicted and actual next tokens (cross-entropy loss) drives billions of tiny weight adjustments via backpropagation and Adam, repeated across trillions of tokens of text, guided by scaling laws that tell researchers roughly how many parameters and tokens to use for a given compute budget.

That process alone yields a base model — an extremely capable but undirected text-continuation engine. Additional stages — supervised fine-tuning on curated instruction examples, then preference optimization (RLHF or DPO) using human judgments of response quality — reshape this raw continuation engine into something that reliably behaves like a helpful, aligned assistant rather than just an autocomplete engine.

Finally, at the moment you actually chat with the model, inference kicks in: tokens are generated one at a time, each new token's Key/Value cached to avoid redundant recomputation, with a sampling strategy (temperature, top-p) deciding how deterministic versus creative each choice should be — and the whole cycle repeats, one token at a time, until the model produces a stop token or reaches a length limit, at which point what started as a raw string of text has become, forty-plus transformer layers and one training pipeline later, a reply.

22. Plain English Recap

Further reading: Vaswani et al., "Attention Is All You Need" (2017); Kaplan et al., "Scaling Laws for Neural Language Models" (2020); Hoffmann et al., "Training Compute-Optimal Large Language Models" (Chinchilla, 2022); Ouyang et al., "Training language models to follow instructions with human feedback" (InstructGPT/RLHF, 2022); Rafailov et al., "Direct Preference Optimization" (2023); Dao et al., "FlashAttention" (2022); Su et al., "RoFormer" (RoPE, 2021).