Transformer Architecture
The mechanics behind every modern LLM.
Explain self-attention from first principles.
Each token is projected into Query, Key, and Value vectors. Attention scores are computed as the dot product of a token's Query with every other token's Key, scaled by 1/√d_k, then passed through softmax to get weights. Those weights are used to combine the Value vectors, so each token's new representation is a weighted mixture of every other token's information - letting the model dynamically decide what to "attend to" regardless of distance.
Why scale by √d_k in attention?
As dimension d_k grows, dot products grow in magnitude, pushing softmax into regions with extremely small gradients (saturation). Dividing by √d_k keeps the variance of the dot products roughly constant regardless of dimension, keeping softmax gradients well-behaved.
What is multi-head attention and why use multiple heads instead of one big one?
Instead of one attention computation over the full embedding dimension, the model splits it into several smaller "heads" that each learn to attend to different types of relationships (syntax, coreference, position, etc.) in parallel, then concatenates and projects the results. This gives more representational diversity than a single attention operation at the same total parameter cost.
Encoder-only vs decoder-only vs encoder-decoder - when is each used?
- Encoder-only (BERT): bidirectional context, best for classification, embeddings, retrieval.
- Decoder-only (GPT, LLaMA, Mistral): causal/autoregressive, generates text left-to-right - the dominant architecture for modern generative LLMs because it scales well and unifies pretraining with generation.
- Encoder-decoder (T5, BART): encoder reads full input bidirectionally, decoder generates conditioned on it - strong for translation and structured seq2seq tasks.
What is causal (masked) self-attention?
A mask is applied to the attention score matrix so each token can only attend to itself and earlier tokens (future positions are set to -infinity before softmax). This enforces autoregressive generation - the model can't "cheat" by seeing future tokens during training, which matches how it must generate at inference time.
Positional encoding: absolute, learned, RoPE, ALiBi - how do they differ?
- Absolute sinusoidal (original Transformer): fixed sin/cos functions added to embeddings; doesn't extrapolate well beyond trained length.
- Learned absolute (GPT-2/BERT): a trainable embedding per position; also struggles beyond max trained length.
- RoPE (Rotary Position Embedding - LLaMA, Mistral, most modern LLMs): rotates Query/Key vectors by an angle proportional to position, encoding relative position directly into the dot product, which generalizes better to longer contexts.
- ALiBi: adds a fixed linear penalty to attention scores based on distance between tokens, no learned parameters, extrapolates well to longer sequences than seen in training.
What is the feed-forward (MLP) block inside a transformer layer doing?
After attention mixes information across tokens, the position-wise feed-forward network (typically expand to 4x hidden dim, apply a non-linearity like GELU/SwiGLU, then project back down) processes each token independently, adding capacity for the model to transform and store learned features. Most of a transformer's parameters live here, and it's widely believed this is where a lot of factual "knowledge" is stored.
What is Mixture-of-Experts (MoE) and why do models like Mixtral use it?
Instead of one dense feed-forward block per layer, MoE has many "expert" feed-forward sub-networks and a router that picks a small subset (e.g., top-2) to activate per token. This decouples total parameter count from compute-per-token: the model can have far more total parameters (more capacity) while only a fraction are used per forward pass, giving better performance per unit of inference compute.
Multi-Head vs Multi-Query vs Grouped-Query Attention - why do they matter for inference?
Standard Multi-Head Attention (MHA) gives every head its own Key/Value projections, which means a large KV cache at inference. Multi-Query Attention (MQA) shares a single Key/Value head across all query heads, drastically shrinking the KV cache and speeding up decoding, at some quality cost. Grouped-Query Attention (GQA) is the middle ground - groups of query heads share a Key/Value head - used in LLaMA-2/3 and Mistral to balance inference speed with quality.
What is SwiGLU and why do modern LLMs use it?
SwiGLU is a gated activation in the feed-forward block: split the projected hidden state, apply Swish to one half, multiply with the other, then project down. It improves quality vs plain GELU-MLP at similar parameter count and is used in LLaMA, PaLM, and most recent open models.