Interview: LLM Metrics and Observability

• Business - task completion rate, CSAT, conversion lift; answers whether the product works for users and revenue.
• Quality - faithfulness, hallucination rate, rubric scores; answers whether answers are correct, grounded, and on-brand.
• System - p95 latency, error rate, GPU utilization; answers whether infrastructure is healthy.
• Cost - tokens per task, $/successful answer, cache hit rate; answers whether spend is efficient.

A healthy stack instruments all four and correlates them on trace_id - system metrics are fast but say nothing about hallucinations, while quality metrics are slow but decisive.

• Correctness - exact match, F1, or unit tests against ground truth.
• Faithfulness - answer claims supported by retrieved context (RAGAS faithfulness, citation checks).
• Relevance - answer addresses the user question (LLM-as-judge, answer relevancy).
• Instruction following - schema validation, format/tone constraints respected.
• Proxy signals - thumbs up/down, regenerate rate, escalation rate (cheap but noisy).

Combine offline golden-set evals (high precision, low freshness) with sampled online scoring (higher freshness, sampling bias risk). Never use a single judge score as the only launch gate.

• Recall@k - fraction of relevant docs in top-k; critical for recall-oriented QA.
• Precision@k - fraction of top-k that are relevant; matters with short context windows.
• MRR - mean reciprocal rank of first relevant hit; good when one correct document is expected.
• nDCG@k - graded relevance with position discounting.
• Hit rate - % queries with at least one relevant doc above threshold; good operational dashboard metric.

Also track embedding latency, index staleness, rerank lift, and context assembly (chunks retrieved vs after dedup, total context tokens). High generation quality with poor retrieval metrics often means the model is hallucinating from parametric memory.

• Empty result rate - queries returning zero chunks above score threshold.
• Low score rate - top-1 similarity below corpus-specific threshold.
• Context length distribution - sudden drops may signal retrieval failure.
• Chunk diversity - redundant chunks waste context budget.
• Rerank lift - mean score delta pre vs post reranker.
• Index lag - time between source doc update and index refresh.

• TTFT (time to first token) - request accepted to first streamed token byte; dominated by prefill (prompt processing). Users perceive this most acutely - target often < 500 ms (small model) to 2 s (large).
• TPOT (time per output token) - median ms between consecutive output tokens after the first; dominated by decode. Typical range 20-80 ms/token depending on hardware.
• E2E latency - full response ready; task-dependent and includes retrieval, tool calls, and post-processing.

Report p50/p90/p95/p99 per span, not averages - LLM latency distributions are heavy-tailed. Optimize perceived speed with streaming even if total E2E is unchanged.

• Track input tokens (system + user + retrieved context + tool results), output tokens, cached input tokens, and often-forgotten embedding/reranker costs separately.
• Compute per-request cost from versioned price tables: (N_in × p_in + N_out × p_out) / 10^6 + C_embed + C_tools.
• Efficiency ratios: tokens per successful task, context efficiency (useful output / total input), retry multiplier, cache savings.
• Guardrails: per-tenant daily caps, max_tokens limits, retrieval top-k caps, agent max_steps at the gateway.

• Steps per session and LLM calls per session - detect runaway loops and correlate with cost.
• Tool calls per session - per-tool latency p95, error rate, timeout rate, retry count.
• Termination reason - success, max_steps, timeout, user_cancel, error.
• Planning proxies - redundant tool calls, dead-end rate, human takeover rate, objective verifier pass rate.

A single user message can span dozens of LLM calls and external API hits. Hard caps on steps, tokens, and tool spend are mandatory production controls - a 30-step agent loop can cost dollars per message.

• Input-side - jailbreak attempt rate, prompt injection in retrieved docs, PII detected in user input, blocked topic rate.
• Output-side - toxicity score distribution, PII leakage rate, over-refusal vs under-refusal rate, policy violation rate.
• Red team cadence - scheduled adversarial suite pass rate; alert on regressions after model or prompt changes.
• Human review queue - flagged volume, time to review SLA, confirmed violation rate (flag precision).

Safety events are sparse - use Wilson score intervals or Bayesian smoothing for rates below 1%. A jump from 0.1% to 2% PII leakage is an incident; small drops may be noise.

• Data drift - input query distribution shifts (language, length, new intents); detect via embedding distance or PSI on features.
• Concept drift - mapping from input to correct answer changes; golden eval regression, feedback drop.
• Upstream model drift - provider silently updates weights; fixed-prompt eval suite, output hash shifts.
• Corpus drift - retrieved docs stale or wrong; index age, source freshness metrics.
• Prompt drift - unversioned prompt edits in prod; config registry hash mismatch alerts.

PSI (Population Stability Index) on binned features: < 0.1 stable, 0.1-0.25 moderate shift, > 0.25 investigate. Correlate drift timestamps with deploy events.

• SLI - measured indicator (e.g., p95 latency < 3 s).
• SLO - target over a window (e.g., 99% of hours meet the SLI).
• SLA - contractual consequence if SLO missed (credits, refunds).

LLM SLOs must cover latency, availability, quality, and cost - not just uptime. Example set: 99.9% non-5xx availability, p95 TTFT < 1.2 s, golden eval pass ≥ 94%, safety red-team pass ≥ 99%, cost per task p50 < $0.02. Use error budgets: when latency budget is exhausted, freeze non-critical deploys and route to faster tiers; when quality budget is exhausted, block prompt experiments and roll back config.

• Top-level - trace_id, span_id, service_name, operation, duration_ms, status, session_id, tenant_id, feature, environment.
• LLM span - model, prompt_version, prompt_hash (not raw PII), input/output tokens, finish_reason, temperature, ttft_ms, estimated_cost_usd.
• RAG span - query_hash, top_k, chunk_ids, scores, embed_model, index_version, hit, context_tokens.
• Agent span - step_index, max_steps, termination_reason, tool.name, tool.latency_ms, tool.status.

Use OpenTelemetry Gen AI conventions where possible. Store hashes and IDs by default; raw prompts/completions go to restricted stores with TTL. Never attach unbounded labels (user_id, full prompt) to Prometheus - use trace backends for high-cardinality detail.

• Executive health - DAU, task success, cost/day, CSAT trend.
• On-call operations - error rate, p95 latency by stage, queue depth, SLO burn rate.
• Quality review - eval scores, feedback rate, drift PSI, slice breakdown.
• Cost FinOps - $ by tenant, tokens/task, cache savings.
• Safety - flag volume, red-team pass, injection rate.
• RAG deep dive - recall@k, empty hits, index lag, rerank lift.

On-call single screen: "Is it up? Slow? Wrong? Expensive?" Use RED (rate, errors, duration) for request paths and USE (utilization, saturation, errors) for GPUs. Limit to 12-15 panels; link every panel to a runbook.

• Counter - monotonic totals: llm_requests_total, llm_tokens_total, llm_errors_total.
• Histogram - latency distributions: llm_request_duration_seconds with buckets aligned to SLO thresholds.
• Gauge - point-in-time: queue depth, inflight requests, eval scores from nightly jobs.
• Bounded labels only: model, feature, tenant_tier, status - never raw user_id or prompt text.

Example PromQL: p95 latency via histogram_quantile(0.95, sum(rate(llm_request_duration_seconds_bucket[5m])) by (le, feature)); error rate as error requests / total requests. Use recording rules for expensive quantiles; export Grafana dashboards as JSON in git.

• P1 page - availability < 99% over 15m, confirmed safety leak spike.
• P2 ticket - p95 latency 2× baseline for 1h, eval score drop 5%.
• P3 info - cost 20% above forecast, moderate PSI drift.

Alert on symptoms users feel: high 5xx rate, latency SLO burn (multi-window 1h + 6h), provider timeout spikes, nightly eval regression, thumbs-down rate > 3σ above 7-day mean, hourly spend > 2× trailing average (agent loops), queue wait p95 > 10 s. Every alert needs runbook_url and owner_team labels; use inhibition to avoid duplicate pages; review alert fatigue monthly.

• Dimensions - tenant/customer, feature/product surface, team/cost center, model tier, environment, prompt version.
• Unit economics - cost per successful task (divide spend by verifier-pass count), cost per active user/month, margin per AI feature.
• Shared costs - GPU cluster split by measured GPU-seconds or token share per feature; avoid equal splits that hide heavy users.
• Tagging discipline - require tenant_id and feature on every internal API call; reject untagged prod calls.

Weekly FinOps review: top 10 expensive tenants/features. Monthly: model routing opportunities (swap traffic to smaller model with < 2% quality loss). Without attribution, LLM bills stay one opaque line item and nobody optimizes.

Never attach unbounded labels (raw user_id, full prompt text, chunk content) to Prometheus metrics - each unique label combination creates a new time series and can explode storage and query cost. Use bounded enums: model, feature, tenant_tier, prompt_version. Store high-cardinality detail (full traces, prompts, chunk IDs) in trace backends like Langfuse, Phoenix, or ClickHouse. A single mis-instrumented label can take down your monitoring stack faster than a model outage.

• Offline - fixed golden sets scored on schedule or in CI; high precision, low freshness, no sampling bias.
• Online - sampled production traffic scored asynchronously (LLM-as-judge); higher freshness but sampling and position bias risk.
• Proxy - user feedback, regenerate rate, edit distance, retry rate; cheap and always-on but noisy and delayed.

Store eval results keyed by config_hash (model + prompt + tools + retrieval settings). Alert when score drops beyond δ on any stratified slice - aggregate averages hide regressions on minority slices that carry legal or revenue risk.