Make Better Decisions by Embracing Uncertainty

Ryne Carbone11 Jun 2024

Problem

Ramp asks applicants to link a bank account so it can set credit limits from recent transactions. Several third-party providers do the linking, and nobody knows in advance which one works best for which bank. Success rates are noisy, differ by financial institution and change over time, and the amount of history per bank varies a lot. A hardcoded provider-per-bank table was not maintainable long-term, and every failed link leaves the customer with manual statement uploads, which a direct link avoids.

Example from the article: a provider with 1 success and 2 failures can still win traffic against a provider with 11 successes and 9 failures, because its estimate is still uncertain.

Solution in brief

Ramp replaced the static table with one Thompson Sampling bandit per financial institution. Each provider's success rate is modeled as a Beta distribution, and every attempt goes to the provider with the highest sampled value. Before launch the team simulated the algorithm under different noise, trend and volume conditions, then measured the rollout with a sequential before/after comparison.

Questions this case answers
  • How to allocate traffic among several interchangeable options for each of thousands of segments when success rates are noisy, drift over time and per-segment A/B tests do not scale
  • How to estimate the effect of replacing a legacy rule-based policy when a parallel randomized test is not justified and a sequential before/after design is the only quick option
  • How to choose the best provider for each of thousands of segments without running a separate A/B test per segment?
  • How to keep routing decisions fresh when success rates drift over time?

Tools and methods

  • Thompson Sampling — routes each attempt by sampling from every provider's success distribution
  • Beta distribution — models the success probability of each provider
  • Monte Carlo simulation — checks convergence speed under different conditions before launch
  • Sequential quasi-experiment — measures the effect against the previous hardcoded policy
  • NumPy — draws Beta samples in the reference implementation
Continue reading — it's free

Google or a one-time email code. No password.