Neural Network & Deep Learning Foundations
The base layer every LLM question builds on.
What is backpropagation and why does it work?
Backpropagation computes gradients of the loss with respect to every parameter using the chain rule, propagating error signals backward from the output layer to the input. It works efficiently because it reuses intermediate gradients computed at later layers instead of recomputing them from scratch for each parameter, making training cost roughly linear in network size rather than exponential.
Why do we need non-linear activation functions?
Stacking linear layers without non-linearity collapses into a single linear transformation, no matter how many layers you add. Non-linear activations (ReLU, GELU, SiLU/Swish) let networks approximate arbitrarily complex functions, which is what gives depth its representational power.
Explain vanishing and exploding gradients, and how modern architectures avoid them.
In deep networks, gradients are products of many partial derivatives; if these are consistently <1 they shrink to zero (vanishing), if >1 they blow up (exploding). Transformers mitigate this with residual/skip connections (gradients have a direct path back), layer normalization, careful weight initialization, and gradient clipping.
What's the difference between batch, layer, and RMS normalization?
- BatchNorm normalizes across the batch dimension per feature - poor fit for variable-length sequences and small batches.
- LayerNorm normalizes across the feature dimension per sample - independent of batch size, standard in early transformers.
- RMSNorm only rescales by the root-mean-square of activations (no mean-centering), which is cheaper and used in most modern LLMs (LLaMA, Mistral) for efficiency with similar stability.
Adam vs AdamW - what's the actual difference?
Adam applies L2 regularization by adding a weight-decay term directly into the gradient before the adaptive moment update, which interacts poorly with Adam's per-parameter scaling. AdamW decouples weight decay from the gradient update, applying it directly to the weights, which gives more predictable regularization and is now the default optimizer for training LLMs.
What is the bias-variance tradeoff and how does it show up in LLM training?
High bias (underfitting) means the model is too simple to capture patterns; high variance (overfitting) means it memorizes training data and fails to generalize. In LLMs this appears as: too few parameters/data → poor benchmark performance (bias); too many epochs on a small fine-tuning set → memorization and loss of general capability (variance), which is why fine-tuning typically uses very few epochs.
What is cross-entropy loss and why is it used for language modeling?
Cross-entropy measures how far the model's predicted token distribution is from the true next token. Minimizing it is equivalent to maximum likelihood estimation: the model learns to assign high probability to tokens that actually appear in the training data. It is the standard loss for both pretraining and SFT.
SGD vs Adam: why do LLMs rarely use plain SGD?
Plain SGD uses one learning rate for all parameters and struggles with sparse, noisy gradients in huge models. Adam adapts per-parameter step sizes using running estimates of gradient mean and variance, converging faster and more stably on transformer training. AdamW is the de-facto default for LLM pretraining and fine-tuning.