How we tested the conversion impact of global payment methods

Parker Zhao, Adam Rinehouse17 Apr 2025

Problem

Stripe wanted to measure how much conversion and revenue a checkout gains from showing payment methods beyond cards. The answer is needed per method, buyer country, device and industry, so the experiment is large. Random assignment is hard because a customer who refreshes or returns must not see a different set of methods. A different set would annoy the shopper and add noise. On Payment Links a new session ID is created on every page load, so the session ID cannot hold the assignment. With 50+ methods and 200 buyer countries, manual analysis would take days.

Example from the article: Stripe withheld random combinations of 50+ payment methods across checkouts.

Solution in brief

Withhold a random combination of payment methods, chosen by a hash of stable request attributes so the same customer always sees the same set. Check sample sizes with a power analysis, an A/A test and a pilot before the real run. Analyze segment effects with a causal forest instead of slicing the data by hand.

Questions this case answers
  • How to randomize a treatment that has many combinations when units have no persistent identifier and must see a consistent experience across repeated visits
  • How to validate sample sizes and the assignment mechanism of a large multi-arm experiment before trusting segment-level results
  • How to estimate heterogeneous treatment effects across millions of segments without manual slicing
  • How to measure the conversion and revenue impact of showing additional payment methods at checkout?

Tools and methods

  • Hash-based assignment — keeps the withheld set stable across refreshes.
  • Power analysis — minimum sample and days needed.
  • A/A test — checks randomization and sample sizes with no treatment.
  • Pilot experiment — confirms implementation and sample counts.
  • Causal forest — estimates conversion uplift per segment.
Continue reading — it's free

Google or a one-time email code. No password.