Lessons Learned From Running Web Experiments

Sneha Vasanth5 Jul 2023

Problem

Square's Ecosystem Discovery team owns high-traffic pages of squareup.com: Homepage, Pricing and Point of Sale. A change there moves metrics for marketing, brand and product teams and for lead generation. Three things go wrong. Primary and secondary metrics disagree and the team argues about the rollout. A visitor changes state mid-test and may leave the experience they started with. A 50/50 launch exposes half the traffic to bugs, slow pages or broken links.

Example from the article: a test that targets only anonymous visitors loses a visitor who signs up during the same session, and that visitor may start seeing the other variant.

Solution in brief

Fix the decision rules before launch. A trade-off matrix maps primary and secondary results to a decision. Sticky bucketing, or targeting all visitors and segmenting afterwards, keeps assignment stable. A phased ramp (5%, 20%, 50%, 100% of traffic) with exit criteria per phase catches problems while few visitors are exposed. Automated analysis and documented learnings keep the process fast.

Questions this case answers
  • How to agree in advance on a ship or no-ship decision when the primary and secondary metrics of an experiment disagree
  • How to limit the damage of an undetected implementation bug when testing a change on a high-traffic page
  • How to keep treatment assignment stable when the targeting attribute of a unit changes during an experiment
  • How to decide whether to roll out a variant when the primary and secondary metrics move in different directions?

Tools and methods

  • A/B test — variant vs control on a web page
  • Trade-off matrix — pre-agreed decision per pattern of metric results
  • Sticky bucketing — keeps a visitor in one variant when their state changes
  • Phased traffic ramp — 5%, 20%, 50% variant share with exit criteria
  • Bonferroni correction — adjusts significance for post-test segments
  • Python automation framework — duration, lift and significance per experiment
Continue reading — it's free

Google or a one-time email code. No password.