Langfuse

What Langfuse Is

Langfuse is observability and LLMOps infrastructure for LLM applications. It captures traces of every request (inputs, outputs, latency, cost, tool calls, retrieval context), lets teams score quality, manage prompt versions, run experiments on datasets, and debug failures in production.

Unlike generic APM, Langfuse understands LLM semantics: token usage, model names, nested agent steps, RAG retrieval spans, and multi-turn sessions. It integrates with LangChain, LangGraph, OpenAI SDK, LiteLLM, LlamaIndex, and custom code via decorators or the REST API.

Refer: Langfuse docs · GitHub

Data Model: Traces, Observations, Sessions

Trace

Top-level unit representing one user-facing request or batch job. Carries user_id, session_id, tags, metadata, and release/environment labels. All nested work hangs under a trace ID.

Observations (spans and generations)

• Generation - LLM call with model, prompt/completion messages, token counts, cost, latency
• Span - non-LLM work: retrieval, tool execution, graph node, HTTP call
• Event - point-in-time log inside a span

Observations form a tree mirroring your call stack. A LangGraph agent run might show: trace > span (graph) > generation (model) > span (tool) > generation (model).

Session

Groups multiple traces for one conversation or workflow (e.g. chat thread over days). Analyze session-level cost, drop-off, and satisfaction scores.

SDK Integration Patterns

Python decorator (framework-agnostic)

from langfuse.decorators import langfuse_context, observe

@observe()
def answer_question(query: str, user_id: str):
    langfuse_context.update_current_trace(user_id=user_id)
    context = retrieve(query)
    return generate(query, context)

@observe(as_type="generation")
def generate(query: str, context: str):
    response = openai_client.chat.completions.create(...)
    return response.choices[0].message.content

@observe auto-creates nested spans. Child calls become child observations. Set as_type="generation" on LLM functions for correct cost attribution.

LangChain / LangGraph callback handler

from langfuse.callback import CallbackHandler

handler = CallbackHandler(
    session_id="chat-9f2a",
    user_id="user_123",
    tags=["production", "rag-v2"],
)

result = chain.invoke(
    {"question": "..."},
    config={"callbacks": [handler]},
)

The handler maps LangChain run IDs to Langfuse observations automatically, including tool calls and retriever steps. Pass the same handler through LangGraph config on graph.invoke.

OpenAI SDK wrapper

from langfuse.openai import openai

completion = openai.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
)

Drop-in replacement import instruments calls without changing call sites. Works with streaming when using supported patterns.

Manual Trace Control

For fine control, use the Langfuse client directly:

from langfuse import Langfuse

langfuse = Langfuse()

trace = langfuse.trace(name="support-ticket", user_id="u1", metadata={"plan": "enterprise"})
retrieval = trace.span(name="retrieval", input={"query": q})
docs = search(q)
retrieval.end(output={"doc_ids": [d.id for d in docs]})

gen = trace.generation(
    name="draft-reply",
    model="gpt-4o",
    input=messages,
    output=reply,
    usage={"input": 1200, "output": 340},
)
trace.update(output=reply)

Manual traces suit non-Python services or when callbacks do not reach all code paths. Flush on shutdown in short-lived workers: langfuse.flush().

Scoring and Quality Feedback

Scores attach to traces, observations, or sessions:

• Numeric - 0 to 1 relevance, latency SLA compliance
• Boolean - thumbs up/down, policy violation yes/no
• Categorical - tone labels, error types

langfuse.score(
    trace_id=trace_id,
    name="user-feedback",
    value=1,
    comment="Accurate citation",
)

Sources of scores

• End-user UI feedback buttons
• Human reviewer queues sampling production traces
• Automated eval scripts (BLEU, exact match, JSON schema pass)
• LLM-as-judge pipelines writing scores back via API

Scores power dashboards, regression alerts, and fine-tuning dataset curation (export low-scored traces as training or preference pairs).

Prompt Management

Langfuse stores versioned prompts with labels (production, staging, latest). Fetch at runtime so copy changes do not require redeploys:

prompt = langfuse.get_prompt("support-agent-system", label="production")
compiled = prompt.compile(customer_name="Acme", tier="enterprise")
messages = compiled  # ready for chat API

Each prompt version tracks config (model hints, temperature), template text, and linked experiments. Compare versions by running the same dataset against v3 vs v4 and measuring scores.

Link prompt versions to generations automatically when using Langfuse-integrated SDKs; traces show exactly which prompt version produced each output.

Datasets and Experiments

Datasets

Collections of input / expected_output items (and optional metadata). Import from CSV, production traces, or manual curation. Datasets are the ground truth for regression testing.

Experiments (dataset runs)

Run a handler function or chain against every dataset item. Langfuse records outputs, latency, cost, and scores per item. Compare runs side-by-side when you change prompts, models, or retrieval settings.

from langfuse import Langfuse

def run_item(*, item, **kwargs):
    return my_rag_chain.invoke({"question": item.input})

dataset = langfuse.get_dataset("golden-qa")
result = dataset.run_experiment(
    name="rag-v2-bge-reranker",
    task=run_item,
)

Use experiments in CI on staging: fail the build if average score drops below threshold vs baseline run.

Cost and Performance Analytics

Generations record model name and token usage. Langfuse computes cost from model price tables (configurable). Dashboards break down:

• Cost per user, feature, model, prompt version
• P50/P95 latency per step and end-to-end
• Error rates and timeout patterns
• Token volume trends for capacity planning

Attribute cost to retrieval vs generation vs tool steps by inspecting span trees. Expensive retriever rerankers often hide inside RAG traces.

Debugging Production Failures

Typical workflow:

1. Filter traces by error tag, low score, or user complaint ID in metadata
2. Open trace timeline: see exact retrieval chunks, tool args, model raw output
3. Compare to a golden trace from datasets
4. Clone inputs into playground, tweak prompt, re-run experiment
5. Ship prompt or retrieval fix; monitor next experiment run

Export trace JSON for support tickets (redact PII first). Session replay shows multi-turn context loss or tool loop bugs.

Langfuse with LangGraph

Pass CallbackHandler in graph config. Each node execution becomes a span; model calls inside nodes become generations. For custom node names in the UI, set run names in node config or use @observe inside node functions.

Long-running graphs: ensure handler flushes periodically in workers. Use trace_id correlation across async continuations after interrupts.

Deployment: Cloud vs Self-Hosted

Langfuse Cloud

Managed SaaS with EU/US hosting, SSO, RBAC, and retention policies. Fastest path for teams without ops overhead.

Self-hosted (Docker / Kubernetes)

Open-source stack: Langfuse web + worker + Postgres + ClickHouse (analytics) + Redis/Blob storage depending on version. Required for strict data residency, air-gapped environments, or cost control at very high volume.

Set LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY in app env. Use separate projects per environment (dev/staging/prod) with different API keys.

Security and Compliance

• Scrub PII before trace.update or use field-level redaction in SDK
• Restrict API keys per environment; never embed secret keys in frontend
• Configure data retention and deletion for GDPR requests
• Role-based access: engineers see traces, annotators see queues, PMs see dashboards
• Audit prompt changes via version history

Langfuse vs LangSmith

Teams often pick Langfuse for open-source/self-host requirements and strong production analytics. LangSmith fits teams all-in on LangChain with tight Studio integration. Both solve tracing and eval; avoid running duplicate tracing in production.

Operational Best Practices

• Sample traces in high-QPS paths (e.g. 10%) but always capture errors and low-confidence paths
• Standardize tags: env, app version, model tier, experiment flag
• Wire user feedback directly to langfuse.score in the API handler
• Schedule weekly dataset runs against production prompts
• Alert on score regression, cost spikes, and p95 latency SLO breaches
• Document which traces are allowed to contain raw prompts vs redacted forms

How the Three Frameworks Fit Together

LangChain composes models, retrievers, and tools. LangGraph runs durable agent graphs with checkpoints and interrupts. Langfuse observes everything: trace each graph invoke, score outputs, version prompts, and regression-test on datasets. This stack is a common production pattern for enterprise RAG and agent products.