Problem
Square's Ecosystem Discovery team owns high-traffic pages of squareup.com: Homepage, Pricing and Point of Sale. A change there moves metrics for marketing, brand and product teams and for lead generation. Three things go wrong. Primary and secondary metrics disagree and the team argues about the rollout. A visitor changes state mid-test and may leave the experience they started with. A 50/50 launch exposes half the traffic to bugs, slow pages or broken links.
Example from the article: a test that targets only anonymous visitors loses a visitor who signs up during the same session, and that visitor may start seeing the other variant.
Solution in brief
Fix the decision rules before launch. A trade-off matrix maps primary and secondary results to a decision. Sticky bucketing, or targeting all visitors and segmenting afterwards, keeps assignment stable. A phased ramp (5%, 20%, 50%, 100% of traffic) with exit criteria per phase catches problems while few visitors are exposed. Automated analysis and documented learnings keep the process fast.
Questions this case answers
- How to agree in advance on a ship or no-ship decision when the primary and secondary metrics of an experiment disagree
- How to limit the damage of an undetected implementation bug when testing a change on a high-traffic page
- How to keep treatment assignment stable when the targeting attribute of a unit changes during an experiment
- How to decide whether to roll out a variant when the primary and secondary metrics move in different directions?
Tools and methods
- A/B test — variant vs control on a web page
- Trade-off matrix — pre-agreed decision per pattern of metric results
- Sticky bucketing — keeps a visitor in one variant when their state changes
- Phased traffic ramp — 5%, 20%, 50% variant share with exit criteria
- Bonferroni correction — adjusts significance for post-test segments
- Python automation framework — duration, lift and significance per experiment
Google or a one-time email code. No password.