LLMOps, Monitoring & Cost
Running LLM systems reliably over time.
What should you log and monitor for an LLM application in production?
- Quality - sampled outputs reviewed against eval criteria over time, user feedback (thumbs up/down), hallucination/faithfulness signals for RAG.
- Performance - latency (time-to-first-token and total), throughput, error rates, timeout rates.
- Cost - tokens in/out per request, cost per request/user/feature.
- Safety - flagged/blocked content rate, jailbreak attempt detection.
- Drift - changes in input query distribution or output quality over time that might signal the model or data has shifted from what was evaluated.
What is prompt/semantic caching and how does it reduce cost?
Exact-match caching stores responses for identical prompts and returns cached results instantly on repeat requests. Semantic caching goes further, checking if a new query is semantically similar (via embedding similarity) to a previously cached query and reusing that response if similarity exceeds a threshold - reducing redundant LLM calls for common/rephrased questions, though it risks returning a stale or slightly-wrong-context answer if the similarity threshold is too loose.
What are guardrails in an LLM system and what layers do they typically operate at?
- Input guardrails - detect prompt injection, PII, off-topic or malicious requests before they reach the model.
- Output guardrails - check generated content for toxicity, PII leakage, policy violations, or format correctness before returning it to the user.
- Structural guardrails - enforce output schemas (e.g. valid JSON matching a contract) so downstream systems don't break on malformed output.
How do you detect and reduce hallucination in production?
- Ground responses in retrieved context (RAG) and instruct the model to answer only from provided context, explicitly allowing "I don't know" as an answer.
- Use a faithfulness/groundedness check (often another LLM call or NLI model) that verifies the answer's claims are actually supported by the retrieved context before returning it.
- Lower temperature for factual tasks to reduce unnecessary creative variance.
- Cite sources so users/downstream systems can verify claims rather than trusting them blindly.
How do you estimate and control the cost of an LLM feature before shipping it?
Estimate expected input/output tokens per request, multiply by expected request volume and the per-token price of the chosen model, and model how this scales with usage growth. Cost controls include: caching, routing simple queries to cheaper/smaller models, capping max output tokens, trimming unnecessary context (e.g. shorter retrieved chunks), and batching non-latency-sensitive workloads to use cheaper batch-processing API tiers where available.
What is data/model drift in the context of an LLM application, and how do you catch it?
Drift is when the distribution of real user inputs (or the "correct" answer for a given input) shifts over time relative to what the system was built and evaluated on - e.g. users start asking about a new product feature the RAG knowledge base doesn't cover yet. You catch it by continuously sampling and reviewing live traffic against your eval criteria, tracking unanswered/low-confidence-response rates over time, and monitoring for spikes in negative user feedback tied to specific query types.