LLM Evals Course: Error Analysis, LLM Judges and RAG Evals
Learn to measure and improve LLM applications with evaluations you can trust. You start with error analysis: reading real traces, open and axial coding, and turning failure modes into an eval dataset. Then you build graders, from code checks and reference metrics to LLM-as-a-judge prompts aligned with human labels through true and false positive rates. The middle of the course covers evaluating RAG retrieval and faithfulness, structured extraction, agents and tool calls, multi-turn conversations, reliability with pass^k, and the statistics of error bars and model comparisons. The last phases cover tooling (Inspect AI, promptfoo, DeepEval and Langfuse), regression gates in CI, online evals and monitoring, human feedback loops, cost and latency trade-offs, reading benchmarks critically, and an eval-driven capstone. Each lesson is a short, hands-on read with tested code and review questions.