LLM Evaluation: Complete Reference

1. Why Evaluation Matters

Large language models produce high-dimensional, stochastic outputs. A single prompt can yield many acceptable answers, and many subtly wrong ones. Without systematic evaluation, teams ship regressions discovered only through user complaints, support tickets, or viral social posts.

Evaluation is the feedback loop that connects model capability to product value. It answers: Is this model or system good enough for our users, on our tasks, under our constraints?

What goes wrong without eval

• Silent regressions: A prompt tweak improves tone but breaks JSON schema compliance on 8% of invoices.
• Leaderboard theater: A model scores well on MMLU but fails your retrieval-grounded support workflow.
• Cost surprises: A "smarter" model doubles tokens per task without improving task success.
• Safety gaps: Jailbreak resistance drops after a fine-tune nobody regression-tested.
• Slice blindness: Average accuracy hides catastrophic failure on low-resource languages or edge-case document formats.

Evaluation spans the full stack

Core principles

1. Task-specific signal beats generic benchmarks for product decisions.
2. Measure what users experience, not only what is easy to automate.
3. Version everything: model ID, prompt hash, retrieval index, tool definitions.
4. Stratify results by category, locale, and user segment.
5. Pair automatic metrics with periodic human review.

Refer: Holistic Evaluation of Language Models (HELM) · AlpacaEval / length-controlled win rates

2. Intrinsic vs Extrinsic Evaluation

Understanding this distinction prevents optimizing the wrong objective.

Intrinsic evaluation

Measures properties of the model in isolation, typically on standardized prompts with reference answers or scoring rubrics. Examples:

• Perplexity on a held-out text corpus
• Accuracy on MMLU, GSM8K, HumanEval
• Win rate on MT-Bench or Arena-style comparisons
• Refusal rate on a red-team prompt set

Intrinsic eval is fast, reproducible, and comparable across models. It is essential for model selection and research, but it only approximates real product performance.

Extrinsic evaluation

Measures performance on the actual downstream task or product the model powers:

• Does the support bot resolve tickets without human escalation?
• Does the coding assistant produce mergeable PRs?
• Does the legal summarizer preserve obligation clauses correctly?
• Does the sales email drafter match brand voice and factual constraints?

Extrinsic eval is slower, noisier, and often requires proprietary data. It is the metric that ultimately matters for business outcomes.

Bridging the gap

Build a custom golden set derived from production tasks. It acts as a proxy extrinsic eval that you can run cheaply in CI. Calibrate it quarterly against true extrinsic outcomes (ticket resolution, revenue, human ratings) to ensure the proxy still correlates.

3. Evaluation Lifecycle

Evaluation is not a one-time gate before launch. It is a continuous lifecycle tied to every change in the ML and product stack.

Phase 1: Define success criteria

• What does "good" mean for users? (correctness, helpfulness, safety, speed, cost)
• What are hard constraints? (JSON schema, regulatory language, PII handling)
• What slices matter disproportionately? (enterprise tier, medical queries, non-English)

Phase 2: Build eval assets

• Curate golden prompts from logs (anonymized, consented)
• Label expected behaviors or reference answers
• Define automatic scorers and human rubrics
• Establish baselines on current production config

Phase 3: Pre-deployment validation

• Run full eval suite on candidate model, prompt, or pipeline
• Compare against baseline with statistical tests where sample size allows
• Review failure cases manually before launch

Phase 4: Controlled rollout

• Canary or A/B test with online metrics
• Monitor safety, latency, cost, and user feedback in real time
• Automatic rollback on threshold breach

Phase 5: Continuous improvement

• Feed production failures into eval set (failure-driven dataset growth)
• Deprecate stale prompts that no longer reflect the product
• Re-baseline after major feature changes

# Eval lifecycle as versioned artifacts
eval_suite/
  golden_v3.2.jsonl      # prompts + labels + metadata
  rubrics/
    support_helpfulness.yaml
    json_schema_invoice.yaml
  baselines/
    2025-06-01_gpt-4o-mini_prompt-f4e8.json
  results/
    2025-07-10_candidate-lora_run-0042.json

4. Benchmark Catalog

Public benchmarks provide coarse capability maps. Use them for model shopping, not as sole ship criteria.

Knowledge and reasoning

Mathematics

Code

Instruction following and chat

RAG and long-context

Safety and robustness

5. Golden Sets and Custom Eval Data

A golden set (eval set, regression suite) is your highest-signal asset: real or realistic prompts with labeled expectations, curated to represent production traffic.

Sourcing strategies

• Production logs: Sample queries (PII-redacted, consent-checked). Highest fidelity.
• Support tickets: Escalated cases become permanent regression items.
• Synthetic augmentation: Paraphrase and perturb real prompts to increase coverage.
• Expert-authored: Domain specialists write edge cases engineers miss.
• Adversarial mining: Red-team failures, jailbreaks, and user-reported bugs.

Golden set schema

{
  "id": "support-0042",
  "prompt": "My order #8821 shows delivered but I never received it.",
  "context": ["chunk-991", "chunk-442"],  // optional RAG refs
  "expected": {
    "must_mention": ["refund", "investigation"],
    "must_not_mention": ["guaranteed refund without verification"],
    "json_schema": null,
    "reference_answer": "..."  // optional
  },
  "metadata": {
    "category": "shipping",
    "locale": "en-US",
    "difficulty": "medium",
    "source": "prod_log_2025-06",
    "human_labeler": "alice@corp.com"
  },
  "grading": {
    "type": "llm_judge",
    "rubric_id": "support_helpfulness_v2"
  }
}

Size and composition guidelines

Quality practices

1. Balance categories to match production mix (or overweight high-risk slices).
2. Version and changelog every addition or label edit.
3. Hold out a secret set that engineers cannot overfit prompts against.
4. Review disagreements between raters; fix ambiguous rubrics.
5. Expire stale items when product behavior intentionally changes.

6. Contamination and Data Leakage

Contamination occurs when benchmark test data (or close paraphrases) appears in pretraining, fine-tuning, or eval-set construction, inflating scores without improving real capability.

Types of leakage

• Pretraining leakage: Benchmark Q&A memorized from the web crawl.
• Fine-tuning leakage: Eval prompts accidentally included in SFT or DPO data.
• Prompt leakage: Engineers iterate prompts against the test set (implicit overfitting).
• Judge leakage: The judge model saw benchmark solutions during training.
• Pipeline leakage: Retrieval index built from documents that contain benchmark answers.

Detection methods

Mitigation playbook

1. Filter training data with decontamination pipelines (e.g., n-gram blocklists from test sets).
2. Keep a private holdout never shown to anyone tuning prompts.
3. Separate dev (for iteration) and test (for reporting) splits.
4. Prefer dynamic benchmarks that rotate or use fresh content.
5. Report contamination audits alongside leaderboard numbers.

# Simple n-gram overlap check (illustrative)
from collections import Counter

def ngrams(text, n=13):
    tokens = text.lower().split()
    return set(tuple(tokens[i:i+n]) for i in range(len(tokens) - n + 1))

bench_ng = ngrams(benchmark_question)
train_ng = ngrams(training_shard)
overlap = len(bench_ng & train_ng) / max(len(bench_ng), 1)
# High overlap -> investigate for contamination

7. Automated Metrics

Automatic metrics scale evaluation beyond what human labelers can afford. Each metric captures a narrow facet of quality; combine several rather than relying on one number.

Classification-style metrics

Text overlap metrics

BLEU and ROUGE measure n-gram overlap against reference summaries. They are cheap and interpretable but correlate weakly with human judgment on open-ended generation. Use them as sanity checks, not primary quality signals.

ROUGE-L measures longest common subsequence overlap. ROUGE-1/2 use unigram/bigram recall.

Semantic similarity metrics

• Embedding cosine similarity between output and reference.
• BERTScore: Token-level contextual similarity.
• BLEURT: Learned metric trained on human ratings.

These capture paraphrase better than n-gram metrics but can reward vague, generic answers that embed similarly to references.

Structured output metrics

• JSON schema validity rate
• Field-level accuracy (per-key exact match)
• SQL execution accuracy (run generated SQL, compare result sets)
• API call validity (correct tool name and argument types)

RAG-specific automatic metrics

• Context precision / recall (are retrieved chunks relevant?)
• Faithfulness / groundedness (is the answer supported by context?)
• Answer relevancy (does the answer address the query?)
• Citation accuracy (do cited spans support claims?)

Operational metrics (always report alongside quality)

• Latency p50 / p95 / p99 per stage
• Tokens in / out per task
• Cost per successful task
• Error rate (timeouts, 5xx, parse failures)

8. LLM-as-Judge

LLM-as-judge uses a strong model to score, rank, or critique outputs against a rubric. It scales subjective evaluation when human labeling is too slow or expensive.

Common judge patterns

Known biases and mitigations

import json
from openai import OpenAI

client = OpenAI()

JUDGE_PROMPT = """You are an expert evaluator.
Score the assistant response on correctness (0-5) and helpfulness (0-5).
Return JSON only: {"correctness": int, "helpfulness": int, "rationale": str}

User query: {query}
Assistant response: {response}
Reference (optional): {reference}
"""

def judge(query, response, reference=""):
    msg = JUDGE_PROMPT.format(query=query, response=response, reference=reference)
    out = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": msg}],
        temperature=0,
        response_format={"type": "json_object"},
    )
    return json.loads(out.choices[0].message.content)

# Calibrate monthly against human labels
# target: Pearson r > 0.7 on core rubric dimensions

Calibration protocol

1. Sample 200+ items with dual human ratings.
2. Run judge on same items; compute correlation and confusion matrix.
3. Adjust rubric wording where judge disagrees systematically.
4. Track judge-model version; re-calibrate when judge upgrades.

9. RAG Evaluation

Retrieval-augmented generation fails in two independent places: retrieval (wrong context) and generation (misuse of context). Evaluate both, plus end-to-end answer quality.

Retrieval metrics

where $\text{rank}_q$ is the position of the first relevant document for query $q$, or 0 if none found.

Generation metrics

• Faithfulness: Every claim in the answer is entailed by retrieved context.
• Answer relevancy: The answer addresses the user question.
• Hallucination under empty context: Does the model abstain or invent when retrieval returns nothing?
• Citation precision: Linked spans actually support the claim.

End-to-end RAG eval with RAGAS

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

# Each row: question, answer, contexts (list[str]), ground_truth (optional)
eval_data = {
    "question": [
        "What is the refund window for enterprise plans?",
        "Who approves SOC2 exceptions?",
    ],
    "answer": [
        "Enterprise refunds are available within 60 days of purchase.",
        "The security team lead approves SOC2 exceptions.",
    ],
    "contexts": [
        ["Enterprise plans include a 60-day refund policy..."],
        ["SOC2 exception requests route to the security team lead..."],
    ],
    "ground_truth": [
        "60-day refund window for enterprise.",
        "Security team lead approves SOC2 exceptions.",
    ],
}

dataset = Dataset.from_dict(eval_data)
result = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

print(result.to_pandas())
# Investigate rows where faithfulness < 0.8 manually

Custom faithfulness checker (LLM-free heuristic + judge)

import re

def extract_claims(answer: str) -> list[str]:
    # Split into sentences as crude claims
    return [s.strip() for s in re.split(r'[.!?]+', answer) if s.strip()]

def claim_supported(claim: str, contexts: list[str]) -> bool:
    # Production: use NLI model or LLM judge with context-only prompt
    ctx = "\n".join(contexts).lower()
    # Heuristic demo: keyword overlap (replace in prod)
    tokens = set(claim.lower().split())
    return len(tokens & set(ctx.split())) >= max(2, len(tokens) // 3)

def faithfulness_score(answer: str, contexts: list[str]) -> float:
    claims = extract_claims(answer)
    if not claims:
        return 1.0
    supported = sum(claim_supported(c, contexts) for c in claims)
    return supported / len(claims)

RAG eval workflow

1. Build labeled (query, relevant_chunk_ids) set from click logs or human annotators.
2. Run retrieval-only eval when changing embeddings, chunk size, or hybrid weights.
3. Run generation eval when changing model, prompt, or reranker.
4. Run full pipeline eval before release; stratify by document type and query length.

10. Agent Evaluation

Agents loop: plan, call tools, observe results, repeat. Evaluation must cover trajectory quality, not only final text.

What to measure

Eval environments

• Sandbox APIs: Mock Stripe, GitHub, CRM with recorded responses.
• Replay buffers: Deterministic tool outputs for regression tests.
• Simulated users: LLM user simulator sends follow-up messages.
• Real staging: Highest fidelity; expensive and flaky.

from dataclasses import dataclass, field

@dataclass
class AgentTrace:
    task_id: str
    steps: list[dict] = field(default_factory=list)
    final_answer: str = ""
    success: bool = False

def score_trace(trace: AgentTrace, gold) -> dict:
    tool_names = [s["tool"] for s in trace.steps]
    expected_tools = set(gold.get("required_tools", []))
    forbidden = set(gold.get("forbidden_tools", []))

    return {
        "task_success": trace.success,
        "tool_recall": len(expected_tools & set(tool_names)) / max(len(expected_tools), 1),
        "forbidden_tool_violation": bool(forbidden & set(tool_names)),
        "step_count": len(trace.steps),
        "within_step_budget": len(trace.steps) <= gold.get("max_steps", 10),
    }

# SWE-bench style: apply patch, run tests, measure resolve rate
# WebArena / OSWorld: UI automation success in simulated environments

Agent benchmark landscape

11. Safety Evaluation

Safety eval measures whether the system refuses harmful requests, avoids toxic outputs, resists jailbreaks, and behaves fairly across demographic slices. It is a release blocker for consumer and enterprise products.

Safety dimensions

Red-team prompt categories

• Direct harmful requests (weapons, malware, harassment)
• Indirect elicitation (hypothetical framing, fictional context)
• Multi-turn escalation (benign start, harmful pivot)
• Tool abuse (SQL injection via agent, unauthorized API calls)
• Multilingual and encoded attacks (ROT13, leetspeak, mixed scripts)

# Safety eval result schema
{
  "prompt_id": "jailbreak-017",
  "category": "indirect_harm",
  "model_response": "I can't help with that...",
  "verdict": "pass",  // pass = safe refusal or safe answer
  "judge": "gpt-4o-safety-rubric-v3",
  "severity": "high"
}

# Report: ASR = attacks_that_succeeded / total_attacks
# Also report over-refusal rate on benign_control_set

Guardrails in the eval loop

Evaluate the full stack, not only the base model:

1. Input classifier (block or route before LLM)
2. System prompt and policy layer
3. Output filter (regex, moderation model)
4. Human review queue for borderline cases

Refer: HarmBench · Safety learning notes

12. Human Evaluation

Humans remain the gold standard for subjective quality, nuanced correctness, and safety edge cases. Automatic metrics should be calibrated against human judgment, not replace it for high-stakes decisions.

When human eval is required

• Major model or prompt launches
• Disputes when automatic metrics and user sentiment diverge
• Rubric design and judge calibration
• Low-incidence, high-severity slices (legal, medical)
• Creative or brand-sensitive outputs

Rating designs

Inter-rater agreement

Track agreement to ensure rubric clarity:

• Cohen's kappa for two raters, categorical labels.
• Fleiss' kappa for multiple raters.
• Krippendorff's alpha for missing data and varied scales.

Target $\kappa > 0.6$ for core dimensions. Low agreement means fix the rubric, not blame raters.

Operational best practices

1. Write concrete rubrics with pass/fail examples (anchor items).
2. Use 2+ raters per item for high-stakes labels.
3. Blind model identity to prevent brand bias.
4. Rotate raters to detect drift and fatigue.
5. Close the loop: thumbs-down in prod feeds labeling queues.
6. Pay fairly and train: qualification batches before production labeling.

13. Online vs Offline Evaluation

Offline eval runs on fixed datasets in a controlled harness. Online eval measures real user interactions in production. Mature teams use both in complementary roles.

Offline evaluation

Online evaluation

Bridging offline and online

1. Correlate offline golden scores with online thumbs-down rate weekly.
2. When correlation drops, refresh golden set from recent failures.
3. Use shadow mode: run candidate model on live inputs, show baseline to user, log both outputs for offline comparison.
4. Use interleaving in ranking contexts (less common for generative LLMs than for search).

# Shadow evaluation (no user impact)
async def handle_request(req):
    baseline = await generate(req, model="prod-gpt-4o-mini")
    shadow = await generate(req, model="candidate-lora")  # async, discard from UI
    log_shadow_pair(req.id, baseline, shadow)
    return baseline  # user always sees production

14. CI Gates and Regression Testing

CI gates block merges or deployments when eval scores drop below thresholds. They turn evaluation from a report into an enforcement mechanism.

Typical gate tiers

Gate criteria examples

• Overall task success $\geq$ baseline - 1% (absolute)
• No slice drops more than 3% (legal, JSON schema, non-English)
• Safety ASR $\leq$ baseline + 0.5%
• p95 latency $\leq$ baseline + 10%
• Cost per task $\leq$ baseline + 5%

# GitHub Actions pseudo-workflow
# eval-gate.yml
on: [pull_request]
jobs:
  llm-eval-smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r eval/requirements.txt
      - run: python -m eval.run --suite smoke --baseline main
      - run: python -m eval.check_regression --max-drop 0.01

Flaky eval handling

LLM outputs are stochastic. Reduce flake:

• Set temperature=0 for regression suites (or fixed seed where supported).
• Run $n=3$ samples for borderline items; majority vote.
• Use confidence intervals on binary pass rates; gate on lower bound.
• Cache model responses for pure prompt-diff PRs when model is unchanged.

15. A/B Testing in Production

A/B tests compare variants (model, prompt, retrieval config) on live traffic with statistical rigor. They are the final extrinsic eval before full rollout.

Experiment design

1. Hypothesis: "Candidate reduces escalation rate without increasing cost."
2. Primary metric: One pre-registered success metric (escalation rate).
3. Guardrail metrics: Safety ASR, latency p95, cost, thumbs-down.
4. Unit of randomization: User ID or session ID (consistent experience).
5. Power analysis: Estimate sample size for detectable effect.
6. Duration: Run through weekly seasonality if possible.

Variant attribution requirements

Every request log must include:

{
  "experiment_id": "exp-2025-07-prompt-v7",
  "variant": "B",
  "model": "gpt-4o-mini",
  "prompt_hash": "a3f91c",
  "retrieval_index_version": "idx-2025-06-14",
  "user_id_hash": "u-8e2f..."
}

Statistical considerations

LLM-specific A/B pitfalls

• Length confound: Variant B is more verbose and looks "better" in ratings.
• Novelty effect: Short-term engagement bump after UI change.
• Heterogeneous treatment effect: Wins on simple queries, loses on complex ones.
• Cost masking: Quality improves 2% but cost rises 40%.

16. Harness Architecture

An eval harness is reproducible infrastructure that runs prompts through your system, scores outputs, and stores results for comparison. Treat it as a first-class service, not a notebook script.

Architecture components

Reference Python harness

"""Minimal but production-shaped LLM eval harness."""
from __future__ import annotations

import hashlib
import json
import time
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Callable

@dataclass
class EvalConfig:
    model: str
    temperature: float
    prompt_template: str
    retrieval_index: str | None = None

    def fingerprint(self) -> str:
        blob = json.dumps(asdict(self), sort_keys=True)
        return hashlib.sha256(blob.encode()).hexdigest()[:12]

@dataclass
class EvalItem:
    id: str
    prompt: str
    metadata: dict
    expected: dict | None = None

@dataclass
class EvalResult:
    item_id: str
    output: str
    scores: dict[str, float]
    latency_ms: float
    error: str | None = None

class EvalHarness:
    def __init__(
        self,
        config: EvalConfig,
        generate_fn: Callable[[str], str],
        scorers: dict[str, Callable[[EvalItem, str], float]],
    ):
        self.config = config
        self.generate_fn = generate_fn
        self.scorers = scorers

    def run_item(self, item: EvalItem) -> EvalResult:
        t0 = time.perf_counter()
        try:
            output = self.generate_fn(item.prompt)
            err = None
        except Exception as e:
            output = ""
            err = str(e)
        latency = (time.perf_counter() - t0) * 1000

        scores = {}
        if not err:
            for name, fn in self.scorers.items():
                scores[name] = fn(item, output)

        return EvalResult(item.id, output, scores, latency, err)

    def run_suite(self, items: list[EvalItem]) -> list[EvalResult]:
        return [self.run_item(it) for it in items]

    def save_run(self, results: list[EvalResult], path: Path, git_sha: str):
        payload = {
            "git_sha": git_sha,
            "config": asdict(self.config),
            "fingerprint": self.config.fingerprint(),
            "results": [asdict(r) for r in results],
            "aggregate": self.aggregate(results),
        }
        path.write_text(json.dumps(payload, indent=2))

    def aggregate(self, results: list[EvalResult]) -> dict:
        ok = [r for r in results if r.error is None]
        if not ok:
            return {"n": len(results), "errors": len(results)}
        keys = ok[0].scores.keys()
        return {
            "n": len(results),
            "errors": len(results) - len(ok),
            "mean_latency_ms": sum(r.latency_ms for r in ok) / len(ok),
            **{f"mean_{k}": sum(r.scores[k] for r in ok) / len(ok) for k in keys},
        }

# --- scorers ---
def exact_match(item: EvalItem, output: str) -> float:
    ref = (item.expected or {}).get("reference_answer", "")
    return float(output.strip().lower() == ref.strip().lower())

def json_valid(item: EvalItem, output: str) -> float:
    try:
        json.loads(output)
        return 1.0
    except json.JSONDecodeError:
        return 0.0

# --- usage ---
# harness = EvalHarness(config, my_generate, {"em": exact_match, "json": json_valid})
# results = harness.run_suite(load_items("golden_v3.jsonl"))
# harness.save_run(results, Path("results/run-0042.json"), git_sha="abc123")

Design requirements

• Deterministic config capture: Every run records model, prompt, tools, index version.
• Parallel execution with rate limiting and retry on 429/5xx.
• Resume support for long suites (checkpoint per item).
• Idempotent run IDs so CI reruns do not duplicate data.
• PII scrubbing before writing outputs to shared storage.

17. Platform Tools and Frameworks

Mature eval platforms add dataset management, experiment tracking, human review UI, and production trace linking. Pick tools that integrate with your observability stack.

Open-source and commercial platforms

Metric and judge libraries

# lm-eval-harness CLI (academic benchmarks)
# pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct \
  --tasks mmlu,gsm8k,ifeval \
  --batch_size auto \
  --output_path results/llama-8b-bench.json

# promptfoo red-team in CI
# promptfoo eval -c promptfooconfig.yaml --filter-first-n 50

Selection criteria

• Does it link production traces to eval datasets?
• Does it support your model providers (OpenAI, Anthropic, vLLM, Bedrock)?
• Can it gate CI with regression thresholds?
• Does it handle PII and access control for enterprise?
• Is there an escape hatch (export JSON, self-host)?

Refer: Langfuse notes · lm-evaluation-harness

18. Eval-Driven Development

Eval-driven development (EDD) mirrors test-driven development: define expected behavior as eval cases first, then iterate prompts, models, and pipelines until the suite passes. It shifts LLM work from artisanal tweaking to engineering discipline.

EDD workflow

1. Capture failure: User report or prod trace becomes a new golden item.
2. Write rubric: Define pass criteria before changing the system.
3. Reproduce: Confirm baseline fails the new item.
4. Fix: Adjust prompt, retrieval, or model.
5. Verify: New item passes; regression suite still passes.
6. Commit: Golden item + fix in same PR.

Culture and process

Relationship to fine-tuning

Eval sets drive training data too:

• Failed golden items become SFT examples with corrected outputs.
• Pairwise preferences from human SBS feed DPO/RLHF.
• Keep train/eval disjoint to avoid fine-tuning leakage.

19. Failure Modes and Debugging

When eval scores drop or users complain, systematic debugging beats random prompt edits. Map symptoms to likely layers.

Symptom to layer map

Common eval system failures

• Metric gaming: Optimizing LLM judge score without improving user outcomes.
• Stale golden set: Tests outdated product behavior; passing CI but failing users.
• Judge drift: Upgraded judge model changes scores without system change.
• Non-deterministic CI: temperature > 0 causes flaky gates.
• Label noise: Ambiguous rubrics produce contradictory human labels.
• Survivorship bias: Eval set lacks adversarial or edge-case prompts.
• Average trap: Mean score hides 0% pass rate on critical slice.

Debugging playbook

1. Reproduce with frozen config (model ID, prompt hash, seed).
2. Inspect retrieved context and tool traces for the failing item.
3. Compare baseline vs candidate side-by-side on failure bucket.
4. Cluster failures by embedding similarity to find patterns.
5. Add 5-10 representative failures to golden set before fixing.
6. Verify fix on failure bucket AND full regression suite.

# Failure clustering sketch
from sklearn.cluster import KMeans
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
failures = load_failed_prompts("run-0042.json")
emb = model.encode(failures)
labels = KMeans(n_clusters=8, random_state=0).fit_predict(emb)
# Review largest cluster first: often one root cause

20. Metric Selection Guide

Choose metrics that match task verifiability, risk level, and budget. This table is the quick reference for designing an eval suite.

Decision flowchart (text)

1. Can you write a programmatic verifier (unit test, SQL, regex, schema)? Use it as primary.
2. Is there a reference answer? EM / F1 + semantic similarity.
3. Is grounding in source docs required? Faithfulness first.
4. Is output subjective? LLM judge calibrated to humans + periodic SBS.
5. Is risk high (legal, medical, safety)? Add human eval and conservative gates.
6. Always report latency, cost, and error rate alongside quality.