Choosing an LLM with Evals: Quality, Cost, Latency
On this page
Should we switch to the new model?
A new model ships every few weeks. Each announcement comes with benchmark charts. The questions for DocsBot are narrower: will it answer Acme's questions better, how much more will it cost per conversation, and will customers wait longer?
Only your eval suite can answer the first question. Your logs answer the other two. Put them together and the decision usually becomes obvious.
Comparing models on your own eval suite
Candidates aren't just models. A smaller model with a better prompt, the same model with fewer retrieved chunks, or a cheap model with a fallback to a larger one are all configurations worth testing. Run each one on the same dataset with the same graders, and record three numbers per configuration:
- quality: the pass rate on your key criteria, with its error bar,
- cost: average tokens per request times the provider's price, per 1,000 requests,
- latency: the 95th percentile time to a complete answer, or to the first token if you stream.
Cost, latency and quality: the Pareto front
A configuration is dominated if another one is at least as good on all three numbers and better on one. Drop those. What remains is the Pareto front: the real trade-offs.
# pareto.py: keep the configurations that no other one beats on quality, cost and latency
configs = [ # hypothetical results from one eval run, for illustration
{"name": "large model, 5 chunks", "quality": 0.93, "cost": 41.0, "p95_s": 6.1},
{"name": "large model, 3 chunks", "quality": 0.92, "cost": 29.0, "p95_s": 5.2},
{"name": "medium model, 5 chunks", "quality": 0.90, "cost": 12.0, "p95_s": 3.4},
{"name": "medium model, 8 chunks", "quality": 0.89, "cost": 16.0, "p95_s": 3.9},
{"name": "small model, 5 chunks", "quality": 0.81, "cost": 3.5, "p95_s": 1.8},
{"name": "small + fallback", "quality": 0.90, "cost": 9.0, "p95_s": 2.9},
]
def dominates(a, b):
no_worse = a["quality"] >= b["quality"] and a["cost"] <= b["cost"] and a["p95_s"] <= b["p95_s"]
better = a["quality"] > b["quality"] or a["cost"] < b["cost"] or a["p95_s"] < b["p95_s"]
return no_worse and better
front = [c for c in configs if not any(dominates(o, c) for o in configs)]
for c in sorted(front, key=lambda c: -c["quality"]):
print(f"{c['name']:<24} quality {c['quality']:.2f} ${c['cost']:>5.1f}/1k p95 {c['p95_s']}s")large model, 5 chunks quality 0.93 $ 41.0/1k p95 6.1s
large model, 3 chunks quality 0.92 $ 29.0/1k p95 5.2s
small + fallback quality 0.90 $ 9.0/1k p95 2.9s
small model, 5 chunks quality 0.81 $ 3.5/1k p95 1.8sTwo configurations drop out. "Medium model, 8 chunks" loses to the 5-chunk version on everything, and the "small + fallback" setup matches the medium model's quality for less money and time. The final choice among the four that remain is a product decision: is one point of quality worth three times the cost? Day 21's error bars matter here, because on a few hundred examples, 0.92 and 0.93 may not be distinguishable at all.
How to read public LLM benchmarks critically
Benchmarks are still useful for building a shortlist. Read them with these questions in mind:
- Does the task resemble yours? A math benchmark says little about retrieval-grounded support answers.
- Is it saturated? When top models all score in the high 90s, differences are mostly noise.
- Could the test data have leaked into training? Public test sets end up on the web, and models trained on the web may have seen them. This is called contamination, and it inflates scores.
- Who ran it, and how? Prompting, number of attempts and grading details change results. Vendor-reported numbers often use settings you won't match.
- Is it a preference leaderboard? Rankings from human votes measure what voters prefer in short chats, which favors style and length as well as correctness.
Then run your own suite. It's the only benchmark built from your users' questions.
Check yourself
Run your own eval suite on it, with the same graders and dataset, and compare quality, cost and latency against the current configuration, with error bars.
B is dominated: A matches its quality and latency for less money, so there's no reason to choose B. Drop it and compare only configurations on the Pareto front, where improving one dimension costs another.
Leaderboards measure different tasks, may be inflated by contamination or favorable settings, and preference votes reward style. None of that checks grounding in Acme's docs.
Check what tasks they measure, whether test data may have leaked into training, and how similar they are to your use case. Then verify on your own evals.
A slightly better model that doubles cost or makes responses noticeably slower can be a worse product. Decisions need all three measured together.