Interview: Evaluation and Benchmarking

Intrinsic evaluation measures properties of the model itself in isolation — perplexity, benchmark accuracy (MMLU, HellaSwag, GSM8K), refusal rate on red-team sets. Extrinsic evaluation measures how well the full system performs on the actual downstream product task (e.g. does the support bot resolve tickets without escalation), which ultimately matters for business outcomes but is slower, noisier, and requires proprietary data. Use intrinsic eval for model shopping; use extrinsic (or a calibrated golden-set proxy) for ship decisions.

Perplexity measures how "surprised" the model is by held-out text — lower means the model assigns higher probability to the actual next tokens, roughly indicating fluency and predictive power. Its limitation is that it doesn't directly measure helpfulness, correctness, or alignment with human preference — a model can have low perplexity on generic text while still being unhelpful, unsafe, or factually wrong on specific tasks. Never use perplexity alone as a product quality signal.

1. Define success criteria — correctness, helpfulness, safety, latency, cost, and hard constraints (JSON schema, regulatory language).
2. Build eval assets — golden prompts from logs, rubrics, automatic scorers, and baselines on current production config.
3. Pre-deployment validation — run full suite on candidates; compare to baseline with statistical tests; manually review failure buckets.
4. Controlled rollout — canary or A/B test with online guardrail metrics and automatic rollback on threshold breach.
5. Continuous improvement — feed production failures into the eval set, deprecate stale items, re-baseline after major changes.

Evaluation is not a one-time gate — it runs on every model, prompt, retrieval, or tool change.

• Sample real (or realistic) production queries — PII-redacted and consent-checked — not just easy synthetic examples.
• Include edge cases, escalated support tickets, and known failure modes deliberately, not only average-case inputs.
• Balance categories to match production mix, or overweight high-risk slices (legal, medical, non-English).
• Version and changelog every addition; keep dev, test, and a secret holdout split to prevent prompt overfitting.
• Pair automated metrics (LLM-as-judge, retrieval recall, schema checks) with periodic human review.
• Size tiers: 20–50 items for PR smoke, 200–1,000 for nightly regression, 1,000+ for major model migrations.

Contamination happens when benchmark test data (or close paraphrases) leaks into pretraining, fine-tuning, prompt iteration, or even the judge model's training corpus — inflating scores without improving real capability. Types include pretraining leakage, fine-tuning leakage (eval prompts in SFT/DPO data), prompt leakage (engineers tuning against the test set), judge leakage, and pipeline leakage (retrieval index containing benchmark answers).

Detection: n-gram overlap against training shards, min-hash dedup, canary strings, perturbation tests (large score drops on rephrased questions suggest memorization), and fresh/private benchmarks (e.g. LiveCodeBench). Mitigation: decontamination filtering, strict dev/test splits, private holdouts, dynamic benchmarks, and reporting contamination audits alongside leaderboard numbers.

• MMLU / MMLU-Pro / GPQA — broad and graduate-level multiple-choice knowledge.
• GSM8K / MATH — grade-school to competition-level math reasoning.
• HumanEval / MBPP / SWE-bench — code generation correctness; SWE-bench tests real GitHub issue resolution.
• HellaSwag / BBH / ARC-Challenge — commonsense and challenging reasoning.
• TruthfulQA — resistance to generating popular misconceptions.
• MT-Bench / AlpacaEval / Arena-Hard — open-ended instruction-following and chat quality via human or LLM-judge preference.
• IFEval — verifiable instruction constraints (word count, JSON format).
• HotpotQA / RAGAS / LongBench — multi-hop QA, RAG faithfulness, and long-context tasks.
• HarmBench / AdvBench — jailbreak resistance and adversarial attack success rate.

• Programmatic first — if you can verify with unit tests, SQL execution, regex, or JSON schema, use that as primary.
• Short-form QA — exact match or token F1; semantic similarity (BERTScore) for paraphrases.
• Structured extraction — schema validity rate plus per-field exact match.
• Code — pass@k with executable unit tests.
• Summarization — ROUGE/BLEU as sanity checks only; LLM judge + faithfulness to source as primary.
• RAG — faithfulness, answer relevancy, context precision/recall, citation accuracy.
• Agents — task success, tool selection accuracy, argument schema compliance, steps to success.
• Always report operational metrics — latency p50/p95, tokens in/out, cost per successful task, and error rate alongside quality.

LLM-as-judge uses a strong LLM to score, rank, or critique outputs against a rubric at a scale human evaluation can't match. Common patterns: pointwise scoring (1–10), pairwise preference (A vs B), binary checklists for CI gates, chain-of-thought judging, and reference-guided comparison.

Pitfalls: position bias (favors first option — mitigate by swapping order), verbosity bias (favors longer answers — use length-controlled prompts), self-preference bias (judge favors its own model family — use a different judge family), anchoring on reference quirks, and sycophancy (agrees with flawed reasoning). Calibrate monthly against 200+ dual-human-rated items (target Pearson r > 0.7); never trust the judge blindly.

G-Eval is a chain-of-thought LLM-as-judge method where the judge first generates evaluation steps for a rubric dimension (e.g. coherence, relevance), then produces a score informed by that reasoning. This structured deliberation typically correlates better with human judgments than a single-shot "rate 1–5" prompt. Use it for subjective tasks like summarization quality or open-ended support responses. Still apply standard judge mitigations: temperature 0, bias corrections, and periodic human calibration — G-Eval reduces but doesn't eliminate judge blind spots.

RAG fails in two independent places — retrieval (wrong context) and generation (misuse of context) — so evaluate both plus end-to-end quality. Retrieval metrics: Recall@k, Precision@k, MRR, nDCG@k (requires relevance labels). Generation metrics: faithfulness (every claim supported by context), answer relevancy, hallucination under empty context, and citation precision.

RAGAS is a framework/library that automates RAG metrics — faithfulness, answer relevancy, context precision, and context recall — using LLM-based scorers on your own corpus. Run it on labeled (question, answer, contexts, ground_truth) rows; investigate rows where faithfulness drops below threshold manually. Workflow: retrieval-only eval when changing embeddings/chunking; generation eval when changing model/prompt; full pipeline eval before release. Debug retrieval first — roughly 40% of RAG failures are retrieval misses, not generation errors.

Agents loop through plan → tool call → observation → repeat, so you must evaluate trajectory quality, not only final text. Key dimensions:

• Task success — goal achieved (binary or partial-credit rubric).
• Tool selection — correct tool rate, unnecessary calls, forbidden tool violations.
• Argument correctness — valid JSON args matching schemas.
• Step efficiency — steps and tokens to success within budget.
• Recovery — handles API errors within N steps.
• Safety — no destructive actions without confirmation.

Use sandbox APIs with replay buffers for deterministic regression tests; log every tool call, observation, and planner decision. Benchmarks like SWE-bench (unit tests pass after patch), WebArena (goal state in simulated sites), GAIA, and τ-bench (multi-turn tool use) provide standardized agent eval. Final-answer-only eval hides compounding errors in the trace.

Safety eval measures refusal of harmful requests, jailbreak resistance, PII leakage, fairness across demographic slices, over-refusal on benign queries, and prompt-injection resistance in RAG docs. Report attack success rate (ASR) on red-team sets and false refusal rate on a benign control set — optimizing only for refusal scores fails the product.

Red-team categories: direct harmful requests, indirect elicitation (hypothetical framing), multi-turn escalation, tool abuse (SQL injection via agent), and encoded/multilingual attacks. Evaluate the full stack — input classifier, system prompt, output filter, and human review queue — not just the base model. Use standardized sets (HarmBench, AdvBench) plus custom attacks from production. Safety regressions are release blockers; gate CI on ASR thresholds for critical slices.

Offline eval runs on versioned golden sets in a controlled harness — reproducible, cheap, safe for destructive agent tests, and fast enough for CI. Weaknesses: dataset staleness, distribution shift from live traffic, and risk of overfitting prompts to the golden set.

Online eval measures real user interactions: thumbs up/down, implicit signals (copy, regenerate, abandon), task outcomes (ticket resolved, code merged), escalation rate, and live latency/cost. Weaknesses: noisy, slower, and requires careful experiment design.

Mature teams use both: offline catches regressions before users see them; online catches what offline missed. Bridge them by correlating offline golden scores with online thumbs-down weekly, refreshing the golden set when correlation drops, and using shadow mode (run candidate on live inputs, show baseline to user, log both for comparison).

CI gates block merges or deployments when eval scores drop below thresholds — turning evaluation from a report into enforcement. Typical tiers:

• PR smoke — 20–50 golden items, 3–8 minutes, every pull request.
• Nightly — 500+ items across all slices, 1–3 hours.
• Pre-release — full golden + safety red-team, 4–12 hours.
• Post-deploy — online guardrail metrics on canary traffic.

Gate criteria examples: overall task success ≥ baseline − 1%; no slice drops more than 3%; safety ASR ≤ baseline + 0.5%; p95 latency ≤ baseline + 10%. Handle stochastic flake: temperature 0, n=3 samples with majority vote on borderline items, confidence intervals on pass rates (gate on lower bound), and cached responses when only prompts change. Anti-pattern: disabling gates after one false alarm — fix statistics instead.

An eval harness is reproducible infrastructure — a first-class service, not a one-off notebook. Core components:

• Dataset loader — versioned golden sets with slice/tag filters.
• Runner — invokes model + RAG + tools with frozen config (model ID, prompt hash, index version).
• Scorer registry — pluggable metrics (exact match, JSON schema, RAGAS, LLM judge).
• Result store — JSON/Parquet per run with git SHA and config fingerprint.
• Comparator — diffs vs baseline, highlights regressions by slice.
• Reporter — dashboards, PR comments, Slack alerts.

Design requirements: deterministic config capture, parallel execution with rate limiting and retry, resume/checkpoint support for long suites, idempotent run IDs, and PII scrubbing before shared storage. Start in-repo with Python; graduate to Langfuse, Braintrust, or W&B when you need UI, dataset collaboration, and cross-team visibility.

• Langfuse / LangSmith — tracing, datasets, evals, prompt management; LangSmith is LangChain-native.
• Braintrust — eval loops, CI integration, human review UI; strong for eval-driven dev.
• Promptfoo — YAML-driven red-team and regression tests in CLI/CI.
• RAGAS — RAG faithfulness, context precision/recall metrics library.
• DeepEval — pytest-style LLM unit tests for developer-local testing.
• lm-eval-harness / inspect-ai — 100+ academic benchmarks and composable agent sandboxes.
• Phoenix (Arize) — embedding drift and trace analysis for RAG debugging.
• W&B — experiment tracking and eval tables for research + production.

Selection criteria: production trace linking, provider support (OpenAI, Anthropic, vLLM, Bedrock), CI gate thresholds, PII/access control, and JSON export/self-host escape hatch.

Eval-driven development mirrors test-driven development: capture a failure as a golden item, write pass criteria before changing the system, confirm baseline fails, fix (prompt/retrieval/model), verify the new item passes and regression suite still passes, then commit the eval item and fix in the same PR. Culture practices: every bug becomes an eval, eval review in PRs, weekly scorecards, eval budget in sprints, and PM-owned success criteria. Failed golden items can become SFT examples or DPO preference pairs — but keep train and eval sets strictly disjoint to avoid fine-tuning leakage. North star: a new engineer breaks production only by failing an eval, not by lacking tribal knowledge about sacred prompts.

Route a fraction of traffic to variant B (new prompt, model, or RAG pipeline), compare a pre-registered primary metric (e.g. escalation rate) and guardrail metrics (safety ASR, latency p95, cost, thumbs-down), and require statistical significance before full rollout. Every request log must include experiment ID, variant, model, prompt hash, retrieval index version, and user/session ID for consistent attribution.

Statistical considerations: avoid peeking (use fixed horizon or sequential testing), correct for multiple comparisons on secondary metrics (Bonferroni/FDR), use cluster randomization for network effects, and CUPED for variance reduction. LLM-specific pitfalls: length confound (verbose variant looks better), novelty effect, heterogeneous treatment effects across query complexity, and cost masking (2% quality gain, 40% cost increase). Ship only when primary metric improves at p < 0.05, guardrails hold, and the effect size justifies operational complexity.

With sufficient sample size, treat pass rates as proportions and compute confidence intervals (Wilson or bootstrap) on the difference between candidate and baseline. Gate on the lower bound of the improvement (or upper bound of degradation) rather than a point estimate — e.g. block merge if lower bound of task-success delta is below −1%. For small golden sets, use McNemar's test on paired items (same prompt, two systems) since items are matched. Account for stochastic outputs: run n=3 samples at temperature 0 where supported and majority-vote per item before computing rates. Report effect size alongside p-values — a statistically significant 0.1% lift may not justify a model swap. Stratify by slice so average gains don't hide catastrophic drops on legal or JSON-schema items.

Map symptoms to layers before random prompt edits:

• Wrong static facts → base model / knowledge cutoff.
• Ignores instructions → prompt / system message (check IFEval, schema rate).
• Answers without doc support → RAG generation (faithfulness, empty-context test).
• Missing relevant info → retrieval (Recall@k, chunk overlap).
• Wrong API called → agent planner (tool selection accuracy).
• Cost spike → inference config (tokens per request, loop detection).
• Jailbreaks after deploy → safety stack (red-team ASR, guardrail logs).
• One locale fails → stratified eval by language slice.

Playbook: reproduce with frozen config, inspect retrieved context and tool traces, compare baseline vs candidate side-by-side on the failure bucket, cluster failures by embedding similarity to find root causes, add 5–10 representative failures to the golden set before fixing, then verify on the bucket AND full regression suite. Watch for eval-system failures too: stale golden sets, judge drift after judge-model upgrades, metric gaming, and the average trap hiding 0% pass on critical slices.