Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 5 · Lesson 25 · 4 min read

Langfuse Datasets and Experiments Tutorial

On this page
  1. Langfuse: closing the loop between production and evals
  2. Building a dataset from production traces
  3. Running an experiment on a Langfuse dataset
  4. Comparing experiment runs
  5. Check yourself

Langfuse: closing the loop between production and evals

Error analysis starts from production traces. Eval datasets grow from the interesting ones. Experiment results send you back to traces to see what changed. If those live in three different tools, someone spends their week copying JSON between them.

Langfuse is an open-source LLM observability platform, self-hosted or cloud, that keeps all three together. Traces come in from your app. You add selected traces to a dataset with one click or one API call. An experiment runs your app over the dataset, scores each item and stores the run, so you can compare it with earlier runs side by side.

Building a dataset from production traces

Code
# dataset.py: create a Langfuse dataset and add items, linked to the traces they came from
from langfuse import get_client

langfuse = get_client()      # reads LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST
langfuse.create_dataset(name="docsbot-dev", description="DocsBot dev split, see Day 4")

langfuse.create_dataset_item(
    dataset_name="docsbot-dev",
    id="pw-reset-01",                                    # stable id: re-running updates, not duplicates
    input="How long is a password reset link valid?",
    expected_output="1 hour",
    metadata={"tags": ["account", "stale_fact"], "plan": "pro"},
    source_trace_id="a1b2c3d4e5f6",                      # the production trace this came from
)

Linking source_trace_id matters more than it looks. Six months from now, when someone asks why this example exists, the original conversation is one click away.

Running an experiment on a Langfuse dataset

An experiment needs a task that runs DocsBot on one item, and evaluators that score the result. Evaluators are plain functions, so the code graders from Day 6 and judges from Day 8 drop straight in:

Python
# experiment.py: run DocsBot over the dataset as a Langfuse experiment
import re
from langfuse import Evaluation, get_client

langfuse = get_client()

def docsbot_task(*, item, **kwargs):
    question = item["input"] if isinstance(item, dict) else item.input
    return answer(question)                              # your DocsBot pipeline

def has_citation(*, input, output, expected_output=None, metadata=None, **kwargs):
    ok = bool(re.search(r"https://docs\.acme\.example/", output))
    return Evaluation(name="has_citation", value=1.0 if ok else 0.0)

def contains_expected(*, input, output, expected_output=None, metadata=None, **kwargs):
    if not expected_output:
        return Evaluation(name="contains_expected", value=None, comment="no reference")
    return Evaluation(name="contains_expected", value=float(expected_output.lower() in output.lower()))

def citation_rate(*, item_results, **kwargs):
    scores = [e.value for r in item_results for e in r.evaluations if e.name == "has_citation"]
    return Evaluation(name="citation_rate", value=sum(scores) / len(scores))

dataset = langfuse.get_dataset("docsbot-dev")
result = dataset.run_experiment(
    name="docsbot-prompt-v2",
    task=docsbot_task,
    evaluators=[has_citation, contains_expected],
    run_evaluators=[citation_rate],                      # one score for the whole run
)
print(result.format())

The item handling covers both cases: dataset items from Langfuse are objects, and you can also pass a plain list of dicts with langfuse.run_experiment(data=[...]) to try things before creating a dataset. Each item's run is traced, so in the UI you can open any scored item and see the full DocsBot trace behind it.

Comparing experiment runs

In the Langfuse UI, a dataset page lists its experiment runs with their scores. Select two runs to compare them item by item: which items changed score, with the outputs side by side. That's the paired view from Day 21, without writing the join yourself. Look at the items that got worse before you look at the averages.

Datasets are versioned by time. get_dataset("docsbot-dev", version=...) fetches the items as they were at a given moment, which lets you rerun an old comparison on exactly the old data.

NoteEvaluators that return None are recorded as skipped. Use that for checks that don't apply to an item, such as a reference comparison when there's no reference, so they don't count as failures.

Check yourself

Creating an item with an existing id updates it instead of adding a duplicate, so your import script can run repeatedly as the source file changes.

Compare the two runs item by item. Averages can hide fixes and breakages that cancel out; the comparison shows exactly which items got worse, with their traces.

It links the example to the real conversation it came from, so anyone reviewing or relabeling it can see the full original context.

Filter traces by score, feedback or tags, review them, and add selected ones as dataset items with their inputs and expected results.

Per-item differences between versions, so you can see exactly which examples improved and which regressed, not just whether the average moved. Two versions with the same average can fail on very different examples, and that's what you need to read.