How Bandit Testing Improves A/B Tests at Wealthfront

Malcolm Craney7 Jun 2022

Problem

Product teams at Wealthfront (a fintech company) test banners and other content with several variants. A classic A/B test keeps all variants at fixed, equal weights until statistical significance is reached, which can take days, weeks or months depending on traffic. During that time a clearly worse variant is shown as often as the best one, and every such impression is a lost conversion. The test measures well, but it does not optimize results while it runs.

Example from the article: a banner test with a control advertising investing and a variant advertising the cash account, both at 50% weight. If the variant performs ten times better, the control still gets half of the traffic until the test is resolved.

Solution in brief

Wealthfront added bandit experiments to its A/B framework. Each day a Spark pipeline counts successes and failures per variant, and a backend job turns them into new variant weights with Thompson sampling. Better variants get more traffic, while a guaranteed weight floor, no re-bucketing and a warm-up period keep the test measurable.

Questions this case answers
  • How to allocate traffic among several variants adaptively, so that conversions lost to weaker variants are limited while each variant still gets enough observations to compare
  • How to retrofit adaptive traffic allocation onto an existing fixed-weight experimentation platform without rebuilding it
  • How to stop sending half of the traffic to a losing variant while a multi-variant test waits for significance?
  • How to add bandit (adaptive) allocation on top of an existing in-house A/B experiment framework?

Tools and methods

  • Thompson sampling — draws from each variant's posterior distribution to set its traffic share
  • Apache Spark — daily pipeline that counts successes and failures per variant from experiment-seen events
  • In-house A/B framework — hosts the bandit as an experiment mode
Continue reading — it's free

Google or a one-time email code. No password.