Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 4 · Lesson 19 · 4 min read

Multi-Turn LLM Evals with Simulated Users

On this page
  1. Why single questions miss multi-turn bugs
  2. Defining a simulated user
  3. Running a simulated conversation
  4. Grading the whole transcript
  5. The limits of user simulators
  6. Check yourself

Why single questions miss multi-turn bugs

Every example so far has been one message in, one answer out. Real support chats aren't like that. The customer gives half the details, DocsBot asks a question, the customer changes their mind, and by turn six DocsBot has forgotten they're on the Starter plan.

These failures need multi-turn tests. Writing full conversations by hand doesn't work well: as soon as DocsBot's reply changes, the scripted next message may no longer make sense. Instead, you script the customer's goal and let a model play the customer.

Defining a simulated user

Each test case is a scenario, not a script:

JSON
{"id": "conv-seat-07", "persona": "busy office manager, short messages, not technical",
 "goal": "add a new colleague who can manage billing", "facts": {"plan": "pro", "colleague": "priya@customer.example"},
 "reveal": "only give the colleague's email when asked",
 "success": "priya added with the billing admin role, customer told how to log in"}

The reveal rule matters. Real users don't give every detail up front, and DocsBot's ability to ask the right question is one of the things you're testing.

Running a simulated conversation

Python
# simulate.py: a model plays the customer until the goal is met or the turn limit hits
import anthropic

client = anthropic.Anthropic()

def simulated_customer(scenario: dict, transcript: list[dict]) -> str:
    system = (
        f"You are a customer of Acme talking to its support assistant. Persona: {scenario['persona']}. "
        f"Your goal: {scenario['goal']}. Facts you know: {scenario['facts']}. Rule: {scenario['reveal']}. "
        "Write only your next message, in character. If your goal is done or you give up, "
        "write [END] on its own line after your message."
    )
    # flip roles: DocsBot's replies are the 'user' turns for the simulator
    flipped = [{"role": "user", "content": "(The chat window opens.)"}] + [
        {"role": "assistant" if m["role"] == "user" else "user", "content": m["content"]}
        for m in transcript]
    response = client.messages.create(model="claude-opus-5-5", max_tokens=500,
                                      system=system, messages=flipped)
    return response.content[0].text

def run_conversation(scenario: dict, docsbot, max_turns: int = 8) -> list[dict]:
    transcript: list[dict] = []
    for _ in range(max_turns):
        message = simulated_customer(scenario, transcript)
        done = "[END]" in message
        transcript.append({"role": "user", "content": message.replace("[END]", "").strip()})
        if done:
            break
        transcript.append({"role": "assistant", "content": docsbot(transcript)})
    return transcript

docsbot is your application, called with the conversation so far. The turn limit stops endless loops, and hitting it is itself a signal: either DocsBot or the simulated customer is going in circles.

Grading the whole transcript

Grade conversations at two levels:

  1. Goal completion. Did the customer get what they came for? For actions, use the end-state checks from Day 17. For questions, a judge reads the transcript and the scenario's success description and returns pass or fail.
  2. Conversation failure modes. Run binary judges for problems that only exist across turns: asked for information already given, forgot an earlier fact such as the plan, contradicted itself, kept going after the customer said goodbye.

Then do what you did on Day 2: read 30 to 50 transcripts and write open-coded notes. Multi-turn failures are less predictable than single-turn ones, and reading is still how you find them.

The limits of user simulators

A simulated customer is another model with its own habits. It's usually more patient, more articulate and more cooperative than real people, and it may "help" DocsBot by volunteering facts it was told to hold back. Check simulator behavior, too: read transcripts specifically for the customer side, and fix scenarios where the simulator broke character. Calibrate against real conversations when you have them, and keep a set of real transcripts replayed up to a key turn as a second kind of test.

Check yourself

A fixed script stops making sense as soon as DocsBot replies differently. A goal lets the simulated customer respond naturally to whatever DocsBot says.

DocsBot may be looping, such as asking for the same information repeatedly, or the simulator isn't recognizing that the goal is met. Read those transcripts to see which.

The test no longer checks whether DocsBot asks for missing details. Fix the simulator prompt and confirm by reading the customer turns.

Check the end state for task success, then judge the conversation for things like asking for information only once and not revealing data before verification.

They're more cooperative and consistent than real people and share the model's blind spots. Validate them against real transcripts and keep real conversations in the dataset.