Catalog My learning Blog
LLM Evals Course: Error Analysis, LLM Judges and RAG Evals · Phase 2 · Lesson 11 · 4 min read

Pairwise LLM Judges: Position and Verbosity Bias

On this page
  1. When pairwise comparison beats grading
  2. LLM judge biases: position, verbosity, self-preference and style
  3. A pairwise judge with position swapping
  4. Reading pairwise results with care
  5. Check yourself

When pairwise comparison beats grading

You're choosing between two system prompts for DocsBot. Both pass the groundedness judge at similar rates. The question now isn't "is this answer acceptable" but "which answer would a customer prefer". That's a pairwise question, and it's often easier for both people and models to answer than an absolute one.

Pairwise judging is how leaderboards like LMArena (formerly Chatbot Arena) rank models from human votes. For your own app it's useful when choosing between prompts or models whose pass rates are close, and for qualities like clarity that resist a pass/fail definition.

LLM judge biases: position, verbosity, self-preference and style

Research on LLM judges, notably the 2023 MT-Bench paper by Zheng and colleagues, documented several systematic biases:

BiasWhat happensControl
PositionThe judge favors the first (or second) answer shownJudge both orders, count only consistent wins
VerbosityLonger answers win even when they add nothingState in the criteria that length isn't a merit; check win rate against length difference
Self-preferenceA model favors outputs that resemble its ownUse a judge model that isn't a candidate; validate against humans
StyleConfident tone, formatting and lists win over plain correct textPut correctness criteria first; grade facts with a separate pass/fail judge

Position bias is the easiest to measure and fix, so do it always.

A pairwise judge with position swapping

Python
# pairwise.py: compare two answers, both orders, count only consistent preferences
import anthropic
from typing import Literal
from pydantic import BaseModel

client = anthropic.Anthropic()

CRITERIA = """You compare two answers from Acme's documentation assistant to the same question.
Prefer the answer a customer would find more useful: it answers the actual question, is
correct according to the documents, and is clear. Length is not a merit; extra text that
doesn't help the customer counts against an answer. If they're equally good, say tie."""

class Preference(BaseModel):
    reasoning: str
    winner: Literal["first", "second", "tie"]

def prefer(question: str, docs: str, first: str, second: str) -> str:
    response = client.messages.parse(
        model="claude-opus-5-5", max_tokens=1000, system=CRITERIA,
        messages=[{"role": "user", "content": f"<question>{question}</question>\n<documents>{docs}</documents>\n"
                   f"<first>{first}</first>\n<second>{second}</second>"}],
        output_format=Preference,
    )
    return response.parsed_output.winner

def compare(question: str, docs: str, a: str, b: str, judge=prefer) -> str:
    """Returns 'a', 'b' or 'tie'. A win counts only if it holds in both orders."""
    one = judge(question, docs, a, b)          # a shown first
    two = judge(question, docs, b, a)          # b shown first
    a_wins = one == "first" and two == "second"
    b_wins = one == "second" and two == "first"
    return "a" if a_wins else "b" if b_wins else "tie"

def win_rate(results: list[str]) -> float:
    """Share of comparisons won by a, counting ties as half."""
    return (results.count("a") + 0.5 * results.count("tie")) / len(results)

Each comparison costs two judge calls. Inconsistent pairs, where the verdict flips with the order, become ties. The share of flipped verdicts is itself a useful number: if it's high, the judge can't tell the answers apart and is mostly reacting to position.

Reading pairwise results with care

  • Report win rate with ties, plus the number of comparisons. A 58 percent win rate over 40 pairs may well be chance (Day 21).
  • Check length. Split results by whether the winner was the longer answer. If longer answers win almost always, your verbosity control isn't working.
  • Keep pass/fail checks alongside. Pairwise tells you which is preferred, not whether either is acceptable. A version that wins 70 percent of comparisons can still fail the groundedness check more often.
  • Align it, too. Have the domain expert make pairwise choices on 50 or so pairs and measure agreement, exactly as on Day 9.

Check yourself

Position bias could explain part or all of the result, since LLM judges often favor the first or second answer. Judge each pair in both orders and count only consistent wins; inconsistent pairs are effectively ties.

Whether wins track length. Compare win rate on pairs where B was shorter, and confirm the criteria say length isn't a merit. Validate with human preferences.

Pairwise only ranks two versions against each other. It can't tell you whether either answer is acceptable, which is what you need for gates and monitoring.

Judges tend to favor answers written by the same model family. Use a different model as judge when comparing models, and validate against human preferences.

For qualities that are hard to define absolutely but easy to compare, like which summary is clearer, especially when choosing between two prompt versions.