How we built it: Payment method A/B testing

Steve Young, Janani Varadharajan12 Nov 2024

Problem

Stripe built a no-code tool for A/B testing which payment methods appear at checkout. Three things make such tests slow or wrong. Each checkout session is one data point, so small merchants wait long for significance. Some payment methods have eligibility rules, so many sessions assigned to treatment look identical to control. For merchants who confirm payments from their own servers, the assignment and the payment cannot be linked by the usual randomization key.

Example from the article: a buy now, pay later option is allowed only for transactions of $50 or more. If most orders are below $50, most "treatment" sessions show the control set of payment methods.

Solution in brief

Randomize by a deterministic hash of UserAgent, IP address and a time window, so one customer can land in different groups over time and each window counts as a data point. Check eligibility of all control and treatment payment methods before assigning, and count only sessions where the two sets differ. Join the confirm event to the render event through the PaymentMethod ID and a session ID.

Questions this case answers
  • How to shorten the time to significance of an experiment when the unit count is small and traffic cannot be increased
  • How to exclude units that could not experience the treatment from an experiment when eligibility depends on several interacting rules
  • How to join an exposure event with an outcome event when the outcome arrives through a path that lacks the randomization key
  • How to get a checkout A/B test to significance faster when the business has few transactions?

Tools and methods

  • A/B test — control vs treatment payment methods, traffic split 0.01% to 99.99%.
  • Hash randomization — UserAgent, IP address and time window mapped to 1–10,000.
  • Eligibility check — keeps only sessions where the two sets differ.
  • Event join pipeline — links confirm events to render events in one table.
Continue reading — it's free

Google or a one-time email code. No password.