Interview: Retrieval-Augmented Generation

Fine-tuning bakes knowledge into weights - expensive to update, prone to hallucination when the model is uncertain, and gives no way to cite sources. RAG retrieves relevant, current documents at query time and feeds them into the prompt as context, so the model can ground answers in verifiable, easily-updatable external knowledge without retraining - and can cite exactly which document supported an answer.

• 1. Ingestion - load and parse source documents.
• 2. Chunking - split documents into retrievable pieces.
• 3. Embedding - convert chunks to vectors with an embedding model.
• 4. Indexing - store vectors (+ metadata) in a vector database.
• 5. Retrieval - embed the incoming query, search for top-k similar chunks (optionally hybrid + filters).
• 6. Reranking (optional) - reorder candidates with a more precise (often cross-encoder) model.
• 7. Generation - inject retrieved chunks into the LLM prompt as context, generate the answer, ideally with citations.

Chunks too small lose context needed to answer the question (a fact split across chunk boundaries); chunks too large dilute relevance (embedding averages over irrelevant content, and you waste context window with noise). Typical starting points are a few hundred tokens per chunk with 10-20% overlap to avoid cutting sentences/facts at boundaries - but the right size is highly content-dependent (dense legal text vs conversational transcripts need very different chunking) and should be tuned against retrieval-quality metrics, not chosen arbitrarily.

Fixed-size chunking splits text every N tokens regardless of content structure, which can cut sentences or ideas in half. Semantic chunking splits at natural boundaries (paragraphs, sections, or points where embedding similarity between consecutive sentences drops significantly, indicating a topic shift), producing chunks that are more self-contained and coherent, generally improving retrieval quality at the cost of more preprocessing complexity.

Initial retrieval (via ANN vector search) is optimized for speed over a huge corpus and uses relatively cheap similarity metrics; a reranker (often a cross-encoder that jointly processes the query and each candidate document, rather than encoding them independently) is more accurate but too slow to run over the full corpus. The standard pattern is retrieve a larger candidate set (e.g. top-50) cheaply, then rerank down to the final top-k (e.g. top-5) with the more accurate model before passing to the LLM.

LLMs tend to attend most reliably to information at the very start and end of their context window, with reduced recall for facts buried in the middle - even when technically within the context limit. This means simply stuffing more retrieved chunks into the prompt doesn't guarantee the model will use the most relevant one; practical mitigations include limiting to fewer, higher-relevance chunks, ordering the most important chunk last (closest to the question), and using reranking to ensure the best chunk isn't buried in the middle.

• Retrieval metrics - recall@k, precision@k, MRR (mean reciprocal rank) against a labeled set of (query, relevant-document) pairs.
• Generation/faithfulness metrics - does the answer actually stay grounded in the retrieved context (faithfulness/groundedness), or does it hallucinate beyond it?
• Answer relevance - does the final answer actually address the user's question, independent of context quality.
• Frameworks like RAGAS and TruLens automate several of these using an LLM-as-judge approach against your own test set.

Raw user queries are often short, ambiguous, or phrased differently from how the answer is written in source documents (vocabulary mismatch). Query rewriting uses the LLM itself to reformulate the query (expand with synonyms, break multi-part questions into sub-queries, resolve pronouns using conversation history) before embedding it for retrieval, improving recall of relevant chunks that a literal match would miss.

Instead of embedding the raw query, HyDE prompts an LLM to generate a hypothetical answer to the question first, then embeds that hypothetical answer and uses it for similarity search. Because the hypothetical answer is stylistically closer to the actual documents you're searching over than a short question is, this often improves retrieval relevance, at the cost of an extra LLM call per query.

Single-pass RAG retrieves once and generates once. Agentic/multi-hop RAG lets the model decide it needs more information, issue additional retrieval queries based on intermediate reasoning (e.g. answer part of a question, realize it needs a follow-up fact, retrieve again), and iterate - useful for complex questions that can't be answered from a single retrieval step, at the cost of higher latency and more LLM calls.

Freshness typically requires an incremental ingestion pipeline (detect new/changed documents, re-chunk and re-embed only what changed, delete stale vectors) rather than full reprocessing. Conflicting information is handled by attaching recency/authority metadata to chunks and instructing the model (or a filtering step) to prefer the most recent/authoritative source, and by surfacing source citations so the end user can judge for themselves.

GraphRAG builds a knowledge graph from documents (entities and relationships) and retrieves subgraphs or community summaries for global questions ("What are the main themes across these reports?"). Plain chunk RAG wins on local factual lookup; GraphRAG helps holistic queries over large corpora.

Before embedding each chunk, prepend a short LLM-generated summary of the parent document so the chunk carries global context in its vector. This reduces retrieval failures when chunks lack surrounding context. Cost: one LLM call per chunk at index time.