Why RAG Exists
LLMs memorize training data imperfectly and cannot know private or post-training facts. RAG retrieves relevant passages at query time and injects them into the prompt so the model answers from supplied evidence. Benefits include reduced hallucination on factual QA, access to proprietary data without full fine-tuning, and easier updates by refreshing the document index instead of retraining weights.
RAG does not guarantee truth. The model can still ignore context, misread passages, or combine retrieved facts incorrectly. Retrieval quality and generation discipline both matter.
Ingestion Pipeline
Documents flow through parsing, cleaning, chunking, embedding, and indexing:
Refer: Original RAG paper · LlamaIndex RAG guide
- Parsing - PDF, HTML, DOCX, tickets, code repos into plain text with structure hints
- Chunking - split into segments (256-1024 tokens typical) with overlap to preserve continuity
- Metadata - title, URL, section, ACL, version, timestamp for filtering and citations
- Embedding - batch encode chunks with a retrieval-tuned model
- Indexing - store in vector DB plus optional sparse index for hybrid search
Chunking strategy is a top quality lever: semantic chunking, heading-aware splits, and parent-child indexes (small chunks for retrieval, large parents for generation context) often beat naive fixed-size windows.
Retrieval Stage
At query time the system embeds the user question (sometimes rewritten), searches the index, and returns top-k chunks. Improvements include:
- Query rewriting - HyDE, multi-query expansion, or LLM-generated sub-questions
- Hybrid search - dense plus BM25 fusion
- Reranking - cross-encoder or LLM reranker on top 50-100 candidates to refine top 5-10
- Metadata filters - scope to user permissions and product area
Generation Stage
Retrieved chunks are packed into the context window with clear delimiters and citation instructions. The system prompt should require grounding: answer only from provided context, cite sources, and refuse when context is insufficient.
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA
vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings())
qa = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model="gpt-4o-mini", temperature=0),
retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
return_source_documents=True,
)
result = qa.invoke({"query": user_question})Context packing must respect token limits. Prioritize highest rerank scores, deduplicate near-identical chunks, and optionally summarize long passages before injection. Lost-in-the-middle phenomenon means models may underweight context placed in the center of very long prompts; put critical chunks near the start or end.
Advanced Patterns
Agentic RAG
The LLM decides when to retrieve, which tools to call, or whether to decompose questions into sub-queries iteratively.
Graph RAG
Knowledge graphs link entities across documents for multi-hop reasoning when flat chunk retrieval misses connections.
Corrective RAG (CRAG)
A grader scores retrieved relevance and triggers re-retrieval or web fallback when chunks are off-topic.
Evaluation and Operations
Measure retrieval (recall@k, MRR) and end-to-end answer quality (faithfulness, citation accuracy) separately. Log retrieved chunk IDs with each response for debugging user reports. Refresh pipelines on schedule and on document change events. Version embedding models and indexes together to avoid mixed-vector search.
Cost control: cache embeddings for static corpora, limit k and reranker calls, and route simple queries to smaller models when retrieval confidence is high.