Problem
A team that ships a production LLM wants to iterate quickly, but the evaluation loop is noisy. LLM judges score identical inputs differently across runs, and LLM-generated references come out as different strings each time they are regenerated. A 2% score change could mean a better model, a drifting judge, shifted references, or a mix. Bugs also stay in production until the next retrain, which takes weeks.
Example from the article: at Airbnb, roughly three-quarters of LLM-generated references differ across labeling runs on identical inputs, and the same judge drifts about 1% across runs on the same dataset, while the real signal is 1–3%.
Solution in brief
Treat it as four layers of systems engineering. Name the noise sources, make evaluation deterministic with a per-sample cache, fix bugs with small scoped LoRA patches validated by that fast evaluation, and run a small representative set through the whole production path before release.
Questions this case answers
- How to separate real model improvement from evaluation noise when both the judge and the reference labels are non-deterministic
- How to ship a narrowly scoped fix to a deployed model quickly without regressing behavior that already works
- How to find defects that appear only at the interfaces between individually validated pipeline components
- How to tell whether a new LLM version is really better when the judge and the reference answers change between runs?
Tools and methods
- Per-sample cache — stores references and judge scores so identical inputs return identical results
- LLM-as-a-judge — scores model outputs, now with stable results across runs
- Micro adapter (small LoRA patch) — a rank below 50 correction layered on an existing adapter
- Canary rollout with automatic rollback — ships each patch like a software hotfix
- Stratified end-to-end test set — traffic-weighted sample plus the tail and past incidents
Google or a one-time email code. No password.