Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 1 · Lesson 1 · 4 min read

What Is an LLM Eval? Why Vibe Checks Fail

On this page
  1. Why vibe checks fail for LLM apps
  2. What an LLM eval is made of: dataset, task, grader, report
  3. Meet DocsBot, the app you'll evaluate
  4. Evals vs unit tests vs benchmarks
  5. Where LLM evals pay off
  6. Check yourself

Why vibe checks fail for LLM apps

You tweak the system prompt so your assistant stops rambling. You try five questions, the answers look tighter, and you ship. A week later support tickets mention that the bot now refuses to answer billing questions. Nobody tried a billing question.

That's a vibe check: reading a handful of outputs and deciding they feel right. It works for the first prototype. It fails once the feature has many kinds of users, a prompt that changes weekly, and a model provider that ships new versions under you. LLM output is open-ended and nondeterministic, so one good answer says little about the next thousand.

An eval replaces the feeling with a measurement. It's a repeatable test that runs your application on a fixed set of inputs and scores the outputs against criteria you wrote down. Think of a driving test with a fixed route and a written checklist: every candidate drives the same streets and is marked on the same points, so two scores can be compared. A vibe check is a friend saying "you seemed fine" after a drive around the block.

What an LLM eval is made of: dataset, task, grader, report

Every eval in this course has the same four parts, whatever tool runs it:

PartWhat it isDocsBot example
DatasetInputs, and sometimes expected outputs120 real customer questions with the correct doc page
TaskThe thing under test, called once per inputThe full DocsBot pipeline: retrieval, prompt, model
GraderCode or a model that scores each output"Cites a page that exists" (code), "answer is supported by the cited page" (LLM judge)
ReportScores you can compare across runsPass rate per failure mode, with an error bar

The dataset and graders are the expensive, valuable parts. The task is just your app. Tools like Inspect AI, promptfoo and DeepEval mostly differ in how you write the other three, and you'll use all of them in the tooling phase.

Meet DocsBot, the app you'll evaluate

All course you'll evaluate one application. DocsBot answers customer questions about Acme's SaaS product. It retrieves pages from the product docs, answers with citations, can look up a customer's plan with a tool, and hands off to a human when it can't help.

It's small enough to understand and has every hard part: retrieval that can miss, answers that can drift from the source, tool calls with arguments, multi-turn chats and refusals. You'll build its eval suite step by step and finish with an eval-driven improvement cycle on Day 30.

Evals vs unit tests vs benchmarks

Two comparisons help place evals:

  • Unit tests check deterministic code and expect 100 percent. An eval suite for an LLM feature rarely hits 100, and that's fine. You track the rate and its direction, and you treat some checks, such as "never reveal another customer's data", as hard gates.
  • Public benchmarks such as MMLU score general model ability on someone else's task. They help choose a model shortlist. They can't tell you whether DocsBot answers Acme's billing questions, which is the only thing your users care about. Day 29 covers reading benchmarks critically.

Where LLM evals pay off

Once you have even a small suite, three things get easier:

  1. Changing prompts and models without fear. Run the suite, compare against the last run, and see what got better and what broke.
  2. Prioritizing work. Failure rates per category show where effort pays most.
  3. Arguing with data. "Hallucinations dropped from 9 to 3 percent on 200 questions" ends debates that "it feels better" starts.

The catch is that a bad eval is worse than none, because it gives false confidence. Graders that reward the wrong thing, datasets that don't look like real traffic, and scores without error bars all mislead. Most of this course is about avoiding those traps.

TipStart with error analysis, not metrics. Generic scores such as "helpfulness 4.1 out of 5" rarely point to a fix. The next lesson shows how to find the failures that matter for your app first.

Check yourself

A fixed dataset that represents real traffic, written criteria, a comparison with the old prompt on the same inputs, and some sense of how much the result could vary by chance.

MMLU measures general knowledge on someone else's questions. It says nothing about retrieval from Acme's docs, citation behavior or handoffs, which only your own eval measures.

The dataset and the graders. The task is your existing app, and the report is mostly arithmetic once the other parts are sound.

A dataset of inputs that look like real traffic, the task that runs your app on them, a grader that decides pass or fail for each output, and a report that summarizes results and lets you read individual failures. Weakness in any one makes the whole eval misleading.

A unit test checks deterministic code against one exact expected result. An eval measures how often a nondeterministic system behaves well across many realistic inputs, so its result is a rate with uncertainty, not a single pass or fail.