When Can LLMs Replace Humans in A/B Tests?

Sebastian Ankargren, Joel Persson, Mårten SchultzbergAug 2026

Problem

Teams want to run an A/B test on an LLM instead of on users: results in hours, no traffic allocation. Randomization identifies the effect by design. A model's answers carry no such guarantee, so the estimated effect can be wrong and shipping decisions follow it. The question is under which conditions LLM outputs identify the human treatment effect.

Example from the article: gpt-4o-mini predicted the click-through rate of each news headline variant in thousands of Upworthy A/B tests. Used as if they were human data, the predictions recovered only 39% of the observed human effect.

Solution in brief

Treat the LLM prediction as a surrogate outcome from biostatistics. It identifies the human effect only if two assumptions hold: surrogacy and comparability. Calibrate the predictions on past human experiments with a flexible model, average several LLM draws per unit, and test the calibration on held-out experiments.

Questions this case answers
  • How to estimate a treatment effect from a cheap model-generated proxy outcome instead of real user outcomes when the proxy systematically shrinks effects
  • How to check that a calibration fitted on past randomized experiments transfers to a new treatment
  • How to predict A/B test outcomes with an LLM without biasing the treatment effect?
  • How to know whether LLM predictions can stand in for human responses in an experiment?

Tools and methods

  • LLM outcome prediction — the model predicts the outcome of a typical user separately for control and treatment.
  • Surrogate endpoint theory — defines when the LLM output can replace the human outcome.
  • Calibration with random forest or gradient-boosted trees — maps LLM predictions to human outcomes on past experiments.
  • Falsification test on held-out experiments — checks the calibrated effect against the known human effect.
  • Averaging several LLM draws per unit — removes sampling-temperature noise.
Continue reading — it's free

Google or a one-time email code. No password.