Crosscheck: Benchmarking AI models in the real world

Nuzhet Atay21 May 2026

Problem

A team that chooses or builds an LLM for a professional use case needs to know which model people in that role prefer. Generic crowdsourced leaderboards average over everyone. Cutting them by role and industry creates three problems. Models are updated during the evaluation, so old votes go stale. Niche segments have a handful of votes, and a model that wins 3 of 3 battles looks unbeatable. Small score gaps get reported as rank differences although they are noise.

Example from the article: in a segment like "legal officers in technology" with dozens of battles, a model that wins 3 of 3 would be ranked first. Run 300 battles and its win rate might settle at 55%.

Solution in brief

Fit a Bradley-Terry model on pairwise human votes for each segment, with three changes. Weight votes by age, add a penalty that pulls scores toward the average, and compute 95% confidence intervals so that only statistically separate models get different tiers. Sample the next model pair by uncertainty, so new models converge faster.

Questions this case answers
  • How to turn pairwise human preference votes into per-segment rankings when items change over time and some segments have very few votes
  • How to report a ranking so that only statistically supported differences between items are shown as distinct ranks
  • How to rank LLMs from pairwise human votes separately for each professional segment?
  • How to keep a model leaderboard current when models are updated during the evaluation?

Tools and methods

  • Bradley-Terry model — turns pairwise wins into one score per model
  • Exponential time decay — recent votes count more, old ones fade
  • Regularization penalty — keeps scores near average when a segment has little data
  • 95% confidence intervals and ordinal tiers — models in one tier are statistically indistinguishable
  • Active sampling of pairs — shows raters the pairs with the most uncertainty
Continue reading — it's free

Google or a one-time email code. No password.