Calibrating LLM-Based Population Estimates with Human Validation

Hiroshi Urata12 Aug 2026

Problem

Teams run an LLM over thousands of records and report the share labeled positive as the share in the population. That share is only an apparent prevalence: false positives push it up, false negatives push it down, and the two rarely cancel. A wrong share can send a sizing decision, such as which investment to fund next, in the wrong direction. Accuracy or F1 from a human check do not say how large the bias is or how to remove it.

Example from the article: Indeed wanted to size how many low-quality job recommendations tied to work-experience issues could be fixed with information the matching system already represents, to choose between improving extraction and building new representations.

Solution in brief

Label a small sample by hand, estimate the LLM's sensitivity and specificity from it, and correct the apparent prevalence with the Rogan–Gladen estimator. Then propagate the uncertainty of all three inputs with the delta method to get a confidence interval, and report the interval rather than a point value.

Questions this case answers
  • How to estimate a population proportion from a cheap but imperfect classifier when only a small gold-labeled sample is affordable
  • How to quantify the uncertainty of a classifier-based proportion that depends on the classifier's own estimated error rates
  • How to estimate the true share of records of some type when an LLM classifies them with errors?
  • How to correct an LLM-based positive rate using a small human-labeled sample?

Tools and methods

  • LLM classification of the full dataset — gives the apparent prevalence Pa
  • Human-validated sample — gives sensitivity and specificity, not only accuracy
  • Rogan–Gladen estimator — corrects Pa for both error types
  • Delta method with binomial variances — standard error and 95% interval of the corrected value
  • Monte Carlo simulation — shows how validation sample size and classifier quality set precision
Continue reading — it's free

Google or a one-time email code. No password.