LLM Metrics and Observability: Complete Reference

1. Metrics Hierarchy

LLM observability spans four layers. Each layer answers a different question and uses different storage, cardinality, and retention policies. Treating all signals as one flat dashboard leads to alert fatigue and blind spots.

Four-layer model

Golden signals adapted for LLMs

Google SRE golden signals (latency, traffic, errors, saturation) still apply, but LLM services add quality and cost as first-class dimensions:

• Latency: time to first token (TTFT), time per output token (TPOT), end-to-end request time.
• Traffic: requests per second, tokens in/out per second, concurrent sessions.
• Errors: HTTP 5xx, provider timeouts, tool failures, schema validation failures.
• Saturation: GPU memory, KV cache pressure, embedding queue backlog, rate-limit headroom.
• Quality: rolling eval scores, thumbs-up rate, escalation rate.
• Cost: spend per feature, per tenant, per successful outcome.

Metric types

Cardinality discipline

Never attach unbounded labels (raw user_id, full prompt text, chunk content) to Prometheus metrics. Use bounded enums: model, feature, tenant_tier, prompt_version. Store high-cardinality detail in trace backends (Langfuse, Phoenix, ClickHouse).

2. Quality Metrics

Quality metrics measure whether model outputs satisfy task requirements. Unlike latency, quality is rarely observable from infrastructure alone. You need eval harnesses, judges, human labels, or proxy signals.

Offline vs online quality

• Offline: fixed golden sets scored on schedule or in CI. High precision, low freshness.
• Online: sampled production traffic scored asynchronously. Higher freshness, sampling bias risk.
• Proxy: user feedback, edit distance, retry rate. Cheap but noisy.

Core quality dimensions

LLM-as-judge metrics

A strong model scores candidate outputs on a 1-5 rubric. Track:

• Mean judge score per prompt version and model.
• Win rate in pairwise A/B (model A vs baseline B).
• Judge-human agreement (calibration set).
• Judge variance across repeated runs (temperature 0 for stability).

Regression detection

Store eval results keyed by config_hash (model + prompt + tools + retrieval settings). Alert when:

for any stratified slice (language, task type, tenant). Aggregate averages hide regressions on minority slices that carry legal or revenue risk.

Quality SLO examples

• Golden-set faithfulness ≥ 0.92 weekly.
• JSON schema pass rate ≥ 99.5% for structured extraction.
• Pairwise win rate vs baseline ≥ 55% on custom eval.

3. Retrieval Metrics

RAG systems fail in retrieval before generation. Split metrics across the retrieval pipeline: embedding, index lookup, reranking, and context assembly.

Classic IR metrics

Production retrieval proxies

When labeled q-d pairs are scarce, monitor proxies:

• Empty result rate: queries returning zero chunks above score threshold.
• Low score rate: top-1 similarity below threshold (tune per corpus).
• Context length distribution: sudden drop may mean retrieval failure.
• Chunk diversity: redundant chunks waste context budget.
• Rerank lift: mean score delta pre vs post reranker.

Embedding and index health

• Embedding latency p95 and error rate.
• Index staleness: lag between source doc update and index refresh.
• Version skew: query embedder version ≠ index embedder version.
• ANN recall vs brute force on sample queries (index degradation).

Context assembly metrics

retrieval_trace = {
  "query_hash": "a9f2",
  "embed_ms": 38,
  "search_ms": 112,
  "rerank_ms": 67,
  "chunks_retrieved": 20,
  "chunks_after_rerank": 5,
  "chunks_after_dedup": 4,
  "total_context_tokens": 1840,
  "top_score": 0.87,
  "hit": true
}

4. Generation Metrics

Generation metrics cover the LLM inference step: decoding behavior, output shape, streaming performance, and post-processing success.

Output structure metrics

Decoding parameters to log

• Model ID and version (provider snapshot date for APIs).
• Temperature, top_p, max_tokens, stop sequences.
• Prompt template version and hash.
• System vs user token counts (context window utilization).

Streaming metrics

• TTFT: time from request start to first token byte.
• TPOT: median ms between consecutive output tokens.
• Stream abort rate: client disconnected before completion.
• Inter-token gap p99: detects batching stalls and preemption.

Logprob and confidence (when available)

Mean token logprob, entropy per step, and calibrated uncertainty help flag low-confidence answers for human review or retrieval fallback. Not all APIs expose logprobs; treat absence as a known gap.

Post-generation pipeline

Track success rates for moderation filters, PII redaction, format repair loops, and tool-call validation. A generation can be "good" yet fail downstream parsing.

5. Latency Metrics

Latency drives user experience and infrastructure sizing. LLM latency is multi-modal: prefill (prompt processing) and decode (token generation) have different scaling behavior.

Key latency definitions

Span breakdown (standard)

latency_ms: {
  "auth": 3,
  "rate_limit_check": 1,
  "embed": 45,
  "retrieve": 120,
  "rerank": 68,
  "prompt_build": 5,
  "prefill": 210,
  "decode": 890,
  "postprocess": 12,
  "total": 1354
}

Percentiles, not averages

Report p50, p90, p95, p99 for each span. Means are misleading because LLM latency distributions are heavy-tailed (long outputs, cache misses, cold starts). SLOs should reference percentiles.

Factors that inflate latency

• Long prompts and large retrieved context (prefill grows).
• High concurrency without continuous batching.
• CPU-bound tokenization on gateway.
• Synchronous tool calls inside the generation path.
• Cross-region provider routing.
• KV cache eviction under memory pressure.

6. Cost and Token Metrics

Token usage is the billing unit for most LLM APIs and the primary driver of self-hosted GPU cost (via throughput and model size). Cost metrics must tie to outcomes, not raw request counts.

Token accounting

Cost estimation

Store price tables per model version. Providers change rates; version your pricing config.

Efficiency ratios

• Tokens per successful task: only count tasks that pass validation.
• Context efficiency: useful answer tokens / total input tokens.
• Retry multiplier: total tokens including failed attempts / first-attempt tokens.
• Cache savings: $ avoided via prompt caching and response caching.

Budget guardrails

Enforce per-tenant daily caps, per-request max_tokens, retrieval top-k limits, and agent max_steps at the gateway. Emit metrics when requests are truncated or rejected by budget policy.

7. Throughput Metrics

Throughput measures how much work the system completes per unit time. For GPUs, the key units are tokens/sec (prefill and decode reported separately) and concurrent sequences.

Service-level throughput

• Requests per second (RPS) by endpoint and priority tier.
• Input tokens/sec and output tokens/sec (aggregate and per GPU).
• Successful tasks per hour (business throughput).
• Batch job completion rate for offline pipelines.

Hardware utilization

Backpressure signals

• Queue depth and wait time growth.
• Rate-limit 429 rate from provider or self-imposed.
• Autoscaling lag (pods pending, cold start frequency).
• Rejected requests due to circuit breaker open.

Throughput and latency trade off via batching. Document the operating point: max RPS at p95 latency target.

8. Agent Metrics

Agents loop: plan, call tools, observe results, repeat. Metrics must capture loop structure, tool reliability, and termination behavior. A single "request" can span dozens of LLM calls.

Agent session metrics

Tool-specific metrics

• Per-tool call count, latency p95, error rate, timeout rate.
• Retry count and idempotency violations.
• Permission denied rate (auth misconfiguration signal).
• Result payload size (large tool outputs blow context).

Planning quality proxies

• Redundant tool calls: same args repeated without state change.
• Dead-end rate: tool returns error or empty, agent continues anyway.
• Human takeover rate: escalation to operator mid-session.
• Task success rate: objective verifier pass (code tests, DB row created).

Multi-agent systems

Track handoff count, message fan-out, supervisor override rate, and per-role token share. Without role-level attribution, you cannot tell whether the planner or executor wastes budget.

9. Safety Metrics

Safety metrics quantify policy compliance, harm prevention, and attack resistance. They complement quality metrics: an answer can be fluent yet unsafe or leak confidential data.

Input-side metrics

• Jailbreak attempt rate (classifier score above threshold).
• Prompt injection detection rate in retrieved documents.
• PII detected in user input (block or redact counts).
• Blocked topic rate (policy categories).

Output-side metrics

Red team and eval cadence

Run scheduled adversarial suites (harmful instructions, injection payloads, multilingual attacks). Track pass rate over time and alert on regressions after model or prompt changes.

Human review queue

• Flagged conversation volume per day.
• Time to human review (SLA).
• Confirmed violation rate (precision of automated flags).
• False negative reports from users (missed unsafe content).

10. User Feedback Metrics

User feedback is the ultimate ground truth for product quality, but it is biased, sparse, and delayed. Treat it as a first-class metric stream linked to traces.

Explicit feedback

Implicit feedback

• Copy rate: user copied answer (positive proxy).
• Regenerate rate: user asked for another answer (negative proxy).
• Edit distance: user changed suggested text before sending.
• Abandon rate: session ended without action after response.
• Follow-up clarification rate: user re-asks similar question.
• Time on response: very short may mean dismiss; very long may mean careful read.

Feedback loop engineering

Every thumbs-down should enqueue: trace replay link, prompt version, model, retrieval chunks, and optional label for eval set. Measure time from feedback to fix deployed as an organizational metric.

Segmentation

Break feedback rates by locale, subscription tier, feature flag, and new vs returning users. A global thumbs-up rate hiding a broken locale is a common failure mode.

11. Drift Metrics

Drift is change in behavior or data distribution that degrades performance without a code deploy. LLM systems drift from model updates, corpus aging, user behavior shifts, and embedding index decay.

Types of drift

Population Stability Index (PSI)

Compare binned distributions of a feature (e.g., query length, retrieval score) between reference and current windows:

PSI < 0.1 stable; 0.1-0.25 moderate shift; > 0.25 investigate.

Operational drift dashboard

• Rolling 7-day eval score vs 30-day baseline.
• Thumbs-up rate trend with confidence bands.
• Mean retrieval top-1 score trend.
• New intent cluster rate (embedding clustering on queries).
• Out-of-vocabulary token rate in user inputs.

Correlate drift timestamps with deploy events: prompt version, model ID change, index rebuild, data pipeline failure.

12. SLOs for LLM Services

Service Level Objectives define measurable targets over a window. Error budgets connect reliability work to product velocity. LLM SLOs must include latency, availability, quality, and cost.

SLI vs SLO vs SLA

• SLI (indicator): measured metric (e.g., p95 latency < 3 s).
• SLO (objective): target over window (e.g., 99% of hours meet SLI).
• SLA (agreement): contractual consequence if SLO missed (refunds, credits).

Example SLO set

Error budget policy

When latency error budget is exhausted: freeze non-critical deploys, enable aggressive caching, route to faster model tier. When quality budget is exhausted: block prompt experiments, roll back last config change, increase human review sampling.

Multi-window alerting

Google SRE recommends burn-rate alerts on multiple windows (e.g., 1h and 6h) to catch fast and slow burns. Apply the same pattern to quality SLOs, not only availability.

13. Trace Schema

A consistent trace schema is the contract between application code, observability backends, and analysts. Use OpenTelemetry conventions where possible; extend with LLM-specific attributes.

Top-level trace fields

{
  "trace_id": "7f3a9c2e1b8d4f6a",
  "span_id": "a1b2c3d4",
  "parent_span_id": null,
  "service_name": "rag-gateway",
  "operation": "chat.completion",
  "start_time": "2026-07-12T10:15:30.123Z",
  "duration_ms": 1456,
  "status": "ok",
  "user_id_hash": "u_8f3e2a",
  "session_id": "sess_9912",
  "tenant_id": "acme_corp",
  "feature": "support_copilot",
  "environment": "production"
}

LLM span attributes

RAG span attributes

• retrieval.query_hash, retrieval.top_k, retrieval.chunk_ids[]
• retrieval.scores[], retrieval.embed_model, retrieval.index_version
• retrieval.hit (boolean), retrieval.context_tokens

Agent span attributes

• agent.step_index, agent.max_steps, agent.termination_reason
• tool.name, tool.latency_ms, tool.status, tool.error_type

Privacy and retention

Store hashes and IDs by default. Raw prompts and completions go to restricted stores with TTL and access audit. GDPR/CCPA deletion must cascade to trace stores by user_id_hash.

Refer: OpenTelemetry Gen AI Semantic Conventions

14. Dashboards

Dashboards translate metrics into decisions. Organize by persona and incident type, not by raw metric names.

Dashboard hierarchy

Design principles

• Every panel links to a runbook or owner.
• Use consistent time ranges and compare-to-yesterday overlays.
• Show percentiles, not only means.
• Include annotation layers for deploys and config changes.
• Limit to 12-15 panels per page; drill-down dashboards for detail.

USE and RED methods

• RED (Rate, Errors, Duration) for request-serving paths.
• USE (Utilization, Saturation, Errors) for GPU and queue resources.

Sample on-call row

Single screen answering: "Is the system up?", "Is it slow?", "Is it wrong?", "Is it expensive?"

1. Availability and 5xx rate (last 1h vs SLO).
2. p95 latency by stage (embed, retrieve, generate).
3. Thumbs-down rate spike detector.
4. Cost per hour vs budget.

15. Prometheus and Grafana

Prometheus pulls or receives time-series metrics. Grafana visualizes them. This stack is the default for self-hosted LLM inference and gateway services.

Instrumenting Python services

from prometheus_client import Counter, Histogram, start_http_server

LLM_REQUESTS = Counter(
    "llm_requests_total",
    "Total LLM requests",
    ["model", "feature", "status"],
)
LLM_LATENCY = Histogram(
    "llm_request_duration_seconds",
    "End-to-end request latency",
    ["model", "feature"],
    buckets=[0.1, 0.25, 0.5, 1, 2, 5, 10, 30],
)
LLM_TOKENS = Counter(
    "llm_tokens_total",
    "Token usage",
    ["model", "direction"],  # direction: input | output
)

def handle_request(model, feature, fn):
    with LLM_LATENCY.labels(model=model, feature=feature).time():
        try:
            result = fn()
            LLM_REQUESTS.labels(model=model, feature=feature, status="ok").inc()
            return result
        except Exception:
            LLM_REQUESTS.labels(model=model, feature=feature, status="error").inc()
            raise

start_http_server(8000)  # /metrics endpoint

PromQL examples

# p95 latency over 5m
histogram_quantile(0.95,
  sum(rate(llm_request_duration_seconds_bucket[5m])) by (le, feature)
)

# Error rate
sum(rate(llm_requests_total{status="error"}[5m]))
  / sum(rate(llm_requests_total[5m]))

# Output tokens per second
sum(rate(llm_tokens_total{direction="output"}[5m])) by (model)

Grafana setup tips

• Use recording rules for expensive quantiles (pre-aggregate in Prometheus).
• Template variables: feature, model, environment.
• Mix Prometheus (system) with ClickHouse or Postgres (quality) via mixed datasources.
• Export dashboards as JSON in git; review changes like code.

Histogram bucket selection

Choose buckets that match SLO thresholds. If p95 SLO is 3 s, include buckets around 1, 2, 3, 5, 10 seconds. Misaligned buckets make quantile estimation noisy.

Exporters for GPU nodes

DCGM exporter for NVIDIA metrics: GPU util, memory, temperature, power. Correlate GPU saturation with queue wait spans.

16. Platform Tools

Specialized LLM observability platforms add trace UI, eval integration, and prompt versioning beyond raw Prometheus. Teams often run Prometheus for SRE plus a LLM platform for debugging.

Tool comparison

Selection criteria

• Self-hosted vs SaaS and data residency requirements.
• Trace schema compatibility (OTel export).
• Eval and human annotation workflow built-in.
• Cost accounting per tenant and feature.
• PII handling and retention policies.
• Integration with your gateway (not only SDK wrapper).

Avoid tool sprawl

Pick one system of record for traces and one for time-series metrics. Duplicate instrumentation across five vendors creates conflicting numbers and double billing.

17. Alerting

Alerts must be actionable, routed to owners, and tied to runbooks. Alert on symptoms users feel, not every metric fluctuation.

Alert tiers

Recommended alert rules

• High error rate: > 1% 5xx for 10m (adjust to traffic volume).
• Latency burn: p95 > SLO threshold for 30m with multi-window confirm.
• Provider degradation: upstream timeout rate > 5%.
• Eval regression: nightly job score < baseline − δ.
• Feedback anomaly: thumbs-down rate > 3σ above 7-day mean.
• Cost runaway: hourly spend > 2x trailing average (agent loops).
• Queue saturation: wait time p95 > 10s for 15m.
• Safety: confirmed PII leakage > 0 in any 1h window.

Alert hygiene

• Every alert has runbook_url and owner_team labels.
• Suppress alerts during planned maintenance windows.
• Use inhibition: do not page latency if availability alert already firing.
• Review alert fatigue monthly: mute or fix noisy rules.

Example Alertmanager route

route:
  receiver: default
  routes:
    - match: { severity: critical }
      receiver: pagerduty
    - match: { team: safety }
      receiver: safety-slack
    - match: { alertname: CostRunaway }
      receiver: finops-email

18. Cost Attribution

Cost attribution assigns spend to teams, features, tenants, and outcomes. Without it, LLM bills appear as a single opaque line item and nobody optimizes.

Attribution dimensions

Unit economics

• Cost per request: simple but ignores success/failure.
• Cost per successful task: divide spend by verifier-pass count.
• Cost per active user (month): product planning metric.
• Margin per AI feature: revenue attributed minus inference cost.

Shared cost allocation

GPU cluster costs split by measured GPU-seconds per feature (from labels on jobs) or by token share. Embedding index storage split by corpus owner. Avoid equal split across teams; it hides heavy users.

Tagging discipline

Require tenant_id and feature on every internal API call. Reject untagged calls in production. Backfill tags from trace_id in batch jobs for legacy paths.

FinOps review cadence

Weekly: top 10 expensive tenants and features. Monthly: model routing opportunities (swap 20% of traffic to smaller model with < 2% quality loss). Quarterly: reserved capacity vs on-demand for self-hosted.

19. Metrics Pipeline (Python)

A production metrics pipeline collects spans from the app, aggregates low-cardinality series to Prometheus, and ships rich events to a warehouse for quality and cost analysis.

Architecture overview

1. App emits OpenTelemetry spans + Prometheus counters/histograms.
2. OTel Collector receives, batches, redacts PII fields.
3. Exporter fan-out: Prometheus scrape endpoint, ClickHouse for traces, S3 parquet for batch eval.
4. Nightly job joins traces with feedback and eval labels.

End-to-end Python example

"""
Minimal LLM metrics pipeline: trace + Prometheus + structured log.
"""
from __future__ import annotations

import hashlib
import json
import time
import uuid
from dataclasses import dataclass, field
from typing import Any

from prometheus_client import Counter, Histogram

# --- Prometheus metrics (low cardinality labels only) ---
REQUESTS = Counter("llm_requests_total", "Requests", ["feature", "model", "status"])
LATENCY = Histogram("llm_latency_seconds", "E2E latency", ["feature"], buckets=[0.5, 1, 2, 5, 10, 30])
TOKENS = Counter("llm_tokens_total", "Tokens", ["model", "direction"])
COST = Counter("llm_cost_usd_micros", "Cost in micro-dollars", ["feature", "model"])

PRICE_PER_1M = {"gpt-4o-mini": {"input": 0.15, "output": 0.60}}


@dataclass
class LLMTrace:
    trace_id: str
    feature: str
    model: str
    prompt_version: str
    input_tokens: int = 0
    output_tokens: int = 0
    spans: list[dict[str, Any]] = field(default_factory=list)
    status: str = "ok"

    def add_span(self, name: str, duration_ms: float, attrs: dict | None = None):
        self.spans.append({
            "name": name,
            "duration_ms": duration_ms,
            "attributes": attrs or {},
        })

    def prompt_hash(self, text: str) -> str:
        return hashlib.sha256(text.encode()).hexdigest()[:16]

    def estimated_cost_usd(self) -> float:
        p = PRICE_PER_1M[self.model]
        return (self.input_tokens * p["input"] + self.output_tokens * p["output"]) / 1_000_000

    def flush_log(self):
        """Ship to structured log / OTel / warehouse."""
        record = {
            "trace_id": self.trace_id,
            "feature": self.feature,
            "model": self.model,
            "prompt_version": self.prompt_version,
            "input_tokens": self.input_tokens,
            "output_tokens": self.output_tokens,
            "cost_usd": self.estimated_cost_usd(),
            "spans": self.spans,
            "status": self.status,
        }
        print(json.dumps(record))  # replace with OTel exporter


def run_rag_request(feature: str, model: str, prompt: str) -> str:
    trace = LLMTrace(
        trace_id=str(uuid.uuid4()),
        feature=feature,
        model=model,
        prompt_version="v3.2",
    )
    t0 = time.perf_counter()
    try:
        # Embed
        t_embed = time.perf_counter()
        time.sleep(0.04)
        trace.add_span("embed", (time.perf_counter() - t_embed) * 1000)

        # Retrieve
        t_ret = time.perf_counter()
        time.sleep(0.11)
        trace.add_span("retrieve", (time.perf_counter() - t_ret) * 1000, {"hit": True, "top_k": 5})

        # Generate
        t_gen = time.perf_counter()
        time.sleep(0.35)
        trace.input_tokens = 1800
        trace.output_tokens = 240
        trace.add_span("generate", (time.perf_counter() - t_gen) * 1000, {
            "prompt_hash": trace.prompt_hash(prompt),
            "finish_reason": "stop",
        })

        answer = "Synthetic answer for metrics demo."
        status = "ok"
    except Exception:
        trace.status = "error"
        status = "error"
        raise
    finally:
        elapsed = time.perf_counter() - t0
        LATENCY.labels(feature=feature).observe(elapsed)
        REQUESTS.labels(feature=feature, model=model, status=trace.status).inc()
        TOKENS.labels(model=model, direction="input").inc(trace.input_tokens)
        TOKENS.labels(model=model, direction="output").inc(trace.output_tokens)
        micros = int(trace.estimated_cost_usd() * 1_000_000)
        COST.labels(feature=feature, model=model).inc(micros)
        trace.flush_log()
    return answer


if __name__ == "__main__":
    run_rag_request("support_copilot", "gpt-4o-mini", "How do I reset my password?")

Pipeline hardening

• Async non-blocking exporters; never block the request path on metric I/O.
• Sampling: 100% errors, 1-10% success traces for high QPS.
• Dead letter queue for failed exports.
• Schema registry for trace JSON evolution.
• Idempotent warehouse loads keyed by trace_id.

20. Master Metrics Catalog

Complete reference of LLM production metrics. Use as an instrumentation checklist when designing or auditing observability.