System Design Case Studies
Common open-ended interview prompts, and how to structure an answer.
"Design a customer support chatbot grounded in our internal docs." What do you cover?
Structure the answer around: (1) ingestion pipeline for docs with a re-indexing strategy for updates, (2) chunking strategy suited to the doc format, (3) hybrid retrieval + reranking, (4) prompt design that forces grounding in retrieved context and allows "I don't know," (5) escalation path to a human when confidence is low, (6) guardrails against off-topic/malicious input, (7) evaluation loop (retrieval recall + faithfulness + user feedback), and (8) cost/latency considerations (model tiering, caching).
"Our RAG system is hallucinating / giving wrong answers. How do you debug it?"
Isolate the failure to retrieval or generation first: check whether the relevant document was actually retrieved (retrieval failure - fix chunking, embedding model, or query rewriting) versus retrieved but ignored/misused by the model (generation failure - tighten the prompt to enforce grounding, check for "lost in the middle" issues, verify chunk ordering, consider a faithfulness-check step). Also check for stale/duplicate/conflicting documents in the index, and confirm the eval set actually reflects real failure cases rather than only easy queries.
"How would you reduce the cost of our LLM feature by 10x without hurting quality much?"
Cover: routing easy queries to a smaller/cheaper model and reserving the large model for hard cases (model cascading), aggressive prompt/context trimming (shorter system prompts, fewer/smaller retrieved chunks), caching (exact and semantic), capping max output length, batching non-real-time workloads, quantizing a self-hosted model, and measuring quality impact of each change against your eval set before rolling out broadly.
"Design an LLM-based content moderation pipeline for user-generated text."
Consider: a fast, cheap first-pass classifier (not necessarily an LLM) to filter obvious cases, an LLM (or fine-tuned classifier) for nuanced/contextual judgment on ambiguous cases, clear policy-grounded prompting/labels rather than vague instructions, human review queue for low-confidence or high-stakes cases, feedback loop from human reviewers back into eval/fine-tuning data, and latency requirements (real-time filtering vs async review) shaping the architecture choice.
"How would you fine-tune a model to adopt a specific brand voice/tone consistently?"
Consider whether prompting alone (a detailed style guide + few-shot examples in the system prompt) is sufficient before reaching for fine-tuning, since it's cheaper and easier to iterate on. If fine-tuning is warranted, curate a few hundred to low-thousands of high-quality (prompt, on-brand-response) pairs, use LoRA/PEFT for efficiency, train for few epochs at low learning rate to avoid catastrophic forgetting of general capability, and evaluate both on-brand-voice adherence and general task quality post-fine-tune.