Learn
Concepts, how things work, pipelines, and engineering tradeoffs. This is your reference material.
Interview Prep
Expandable Q&A for revision and interviews. Separate from the learning notes.
Foundations and Architecture
- Foundations
Neural networks, loss, optimizers, normalization. - Transformers
Attention, blocks, RoPE, MoE, GQA. - Tokenization
Subword tokenizers and vocabulary design. - LLMs
Large models, scaling, lifecycle, open vs API. - SLMs
Small models, edge, distillation, routing.
Training Pipeline
- Pretraining
Data, objectives, scaling laws, stability. - Distributed Training
Parallelism, ZeRO, FSDP, clusters. - Fine-Tuning
SFT, LoRA, QLoRA, adapters. - Post-Training
RLHF, DPO, alignment, preferences.
Prompting and Retrieval
- Prompt Engineering
System prompts, CoT, sampling, injection. - Embeddings
Vectors, contrastive training, hybrid search. - Vector Databases
ANN indexes, tuning, product comparison. - RAG
Ingestion, chunking, retrieval, generation. - Prompt Compression
LLMLingua, Headroom, when to compress, pros and cons.
Inference, Deployment, Operations
- Inference Optimization
KV cache, quantization, batching. - Deployment
Serving stacks, scaling, gateways. - LLMOps and Monitoring
Logs, guardrails, cost, drift. - Evaluation
Benchmarks, metrics, eval sets.
Agents and Protocols
Frameworks
Safety and System Design
- Safety
Hallucination, jailbreaks, bias, PII. - System Design
End-to-end AI product architecture.