Problem
A team runs an A/B pilot of an offer and gets no decision: the effect is near zero and the confidence interval is wide. The business wants an answer. Repeating the test is the cleanest option but costs as much as the pilot or more. Slicing segments after the fact is quick, but a post-hoc signal is only a hypothesis. The compromise is a narrower targeting rule, and several candidates soon compete. An A/B test for each does not scale.
Example from the article: the pilot "did not lose, it just did not pick a side". The team builds a targeting policy B and asks whether it deserves a narrower repeat pilot.
Solution in brief
Use off-policy evaluation (OPE): estimate the value of the new policy B from the logs of the old policy A, with uncertainty and data diagnostics. The estimate is only as trustworthy as the overlap between what A did and what B would do, so the report shows overlap, effective sample size and weight tails next to the delta.
Questions this case answers
- How to compare a new decision policy with the current one from logs of the current policy before an online test, when the logs cover the new policy's actions only partially
- How to decide what to do after an underpowered test with a near-zero effect and a wide interval, and how to screen many candidate policies without a test for each
- How to compare a new targeting policy with the old one before running an A/B test
- How to decide what to do after an A/B pilot with a near-zero effect and a wide confidence interval
Tools and methods
- Off-policy evaluation (OPE) — value of a never-run policy from another policy's logs
- Replay, IPS / SNIPS, Direct Method, Doubly Robust, Switch / clipping — estimators
- Support diagnostics: overlap, ESS, weight tails — is there evidence about B in the logs
- OffPolicyLab — the speaker's prototype library for the whole flow and the report
Google or a one-time email code. No password.