Bootstrap Confidence Intervals for LLM Evaluation

James Fiedler8 Jul 2026

Problem

LLM accuracy on a test set of N inputs hides two sources of noise: which inputs were sampled and how the model happened to answer, since one input can give different outputs on different calls even at temperature zero. An interval that ignores the second source is too narrow. Repeating the model k times per input adds data, but runs on one input are correlated, so treating them as independent is wrong too. Either mistake distorts the verdict on which prompt or model is better.

Example from the article: comparing chain-of-thought (CoT) and direct-answer prompts on MMLU-Pro math and biology, where gaps of a few points must be told apart from run-to-run noise.

Solution in brief

Run each input k times and treat the input with its k runs as one cluster. Resample whole clusters with replacement, recompute the metric, and take the 2.5th and 97.5th percentiles for a 95% interval. In the authors' simulations it kept coverage near 95% with the narrowest intervals of five strategies.

Questions this case answers
  • How to quantify the uncertainty of a metric when every unit is measured several times by a noisy process and the repeats are correlated within the unit
  • How to split a fixed measurement budget between more units and more repeats per unit
  • How to put a confidence interval around LLM accuracy when the same input gives different outputs on different runs?
  • How to use k repeated LLM runs per input to get a tighter interval without biasing the estimate?

Tools and methods

  • Cluster bootstrap — interval from resampled inputs that carry all their k runs
  • Intraclass correlation ρ and variance formula — predicts the gain from extra runs
  • Paired cluster bootstrap — interval for the difference between two systems
  • BCa correction — bias and skew adjustment for nonlinear metrics
  • OpenRouter — API used to run the models in the MMLU-Pro study
Continue reading — it's free

Google or a one-time email code. No password.