From weeks to a day: how we made LLM evaluation fast enough to iterate on

Baharak Saberidokht14 Jul 2026

Problem

A team that ships a production LLM wants to iterate quickly, but the evaluation loop is noisy. LLM judges score identical inputs differently across runs, and LLM-generated references come out as different strings each time they are regenerated. A 2% score change could mean a better model, a drifting judge, shifted references, or a mix. Bugs also stay in production until the next retrain, which takes weeks.

Example from the article: at Airbnb, roughly three-quarters of LLM-generated references differ across labeling runs on identical inputs, and the same judge drifts about 1% across runs on the same dataset, while the real signal is 1–3%.

Solution in brief

Treat it as four layers of systems engineering. Name the noise sources, make evaluation deterministic with a per-sample cache, fix bugs with small scoped LoRA patches validated by that fast evaluation, and run a small representative set through the whole production path before release.

Questions this case answers
  • How to separate real model improvement from evaluation noise when both the judge and the reference labels are non-deterministic
  • How to ship a narrowly scoped fix to a deployed model quickly without regressing behavior that already works
  • How to find defects that appear only at the interfaces between individually validated pipeline components
  • How to tell whether a new LLM version is really better when the judge and the reference answers change between runs?

Tools and methods

  • Per-sample cache — stores references and judge scores so identical inputs return identical results
  • LLM-as-a-judge — scores model outputs, now with stable results across runs
  • Micro adapter (small LoRA patch) — a rank below 50 correction layered on an existing adapter
  • Canary rollout with automatic rollback — ships each patch like a software hotfix
  • Stratified end-to-end test set — traffic-weighted sample plus the tail and past incidents
Continue reading — it's free

Google or a one-time email code. No password.