The New Policy Hasn't Run Yet: Comparing It to the Old One Before an A/B Test

Ivan Yurashku23 May 2026

Problem

A team runs an A/B pilot of an offer and gets no decision: the effect is near zero and the confidence interval is wide. The business wants an answer. Repeating the test is the cleanest option but costs as much as the pilot or more. Slicing segments after the fact is quick, but a post-hoc signal is only a hypothesis. The compromise is a narrower targeting rule, and several candidates soon compete. An A/B test for each does not scale.

Example from the article: the pilot "did not lose, it just did not pick a side". The team builds a targeting policy B and asks whether it deserves a narrower repeat pilot.

Solution in brief

Use off-policy evaluation (OPE): estimate the value of the new policy B from the logs of the old policy A, with uncertainty and data diagnostics. The estimate is only as trustworthy as the overlap between what A did and what B would do, so the report shows overlap, effective sample size and weight tails next to the delta.

Questions this case answers
  • How to compare a new decision policy with the current one from logs of the current policy before an online test, when the logs cover the new policy's actions only partially
  • How to decide what to do after an underpowered test with a near-zero effect and a wide interval, and how to screen many candidate policies without a test for each
  • How to compare a new targeting policy with the old one before running an A/B test
  • How to decide what to do after an A/B pilot with a near-zero effect and a wide confidence interval

Tools and methods

  • Off-policy evaluation (OPE) — value of a never-run policy from another policy's logs
  • Replay, IPS / SNIPS, Direct Method, Doubly Robust, Switch / clipping — estimators
  • Support diagnostics: overlap, ESS, weight tails — is there evidence about B in the logs
  • OffPolicyLab — the speaker's prototype library for the whole flow and the report
Continue reading — it's free

Google or a one-time email code. No password.