Method

Power analysis / MDE

Choosing sample size and minimum detectable effect before a test.

28 cases from 19 companies apply Power analysis / MDE. 8 were checked in an online A/B test; 4 report results in production.

Typical tasks: Get experiment results faster, Avoid false wins in A/B tests, Measure impact without an A/B test, Experiment when users affect each other and 3 more. Companies: DoorDash, Spotify, Instacart, Expedia Group, Indeed, Pinterest and 13 more. Often combined with: A/B test, Geo experiment, Bootstrap, A/A test, Stratified sampling / assignment and 35 more.

Bootstrap Confidence Intervals for LLM Evaluationgeneral2026-07
From Eight Hours to Eight Minutes: Automating A/B Test Analysis on Databricksgaming2026-06
LLM-Powered Relevance Assessment for Pinterest Searchsocial2025-12
How DoorDash Ads keep consumers first with budget A/B experimentationadtech2025-09
Beyond Winning: Spotify's Experiments with Learning Frameworkmedia-streaming2025-09
Speed with Rigor: testing smarter with group sequential designclassifieds2025-08
How we tested the conversion impact of global payment methodsfintech2025-04
Introducing the new A/B testing estimatorgeneral2025-01
Unlocking 66% Faster Experiments with Covariate Adjustment: Power Analysis for Clustered Data, Ratio Metrics, and OLS Regressionmarketplace2024-06
So A/B Tests Are Just OLS Regression?general2024-05
Sharpening the Blur: Removing dilution to maximize experiment powerfood-delivery2024-05
Measuring Marketing Success: The Power of Incrementality and Geo Testingtravel2024-05
Fixed-Power Designs: It’s Not IF You Peek, It’s WHAT You Peek atmedia-streaming2024-05
Measuring Marketing Effectiveness in a Cookie-Less Worldsaas-b2b2024-04
Grab Experiment Decision Engine - a Unified Toolkit for Experimentationmarketplace2024-04
Milestones on our Journey to Standardize Experimentation at The New York Timesgeneral2024-03
Risk-Aware Product Decisions in A/B Tests with Multiple Metricsmedia-streaming2024-03
Experiment Faster and with Less Effortfood-delivery2024-02
Improving the OpenChat recommendation model through offline and online A/B testingsocial2024-01
Privacy Sandbox Market Testing: a statistical model for measuring performance impactsadtech2023-12
Experimentation at Spotify: Three Lessons for Maximizing Impact in Innovationmedia-streaming2023-08
Leveraging Switchbacks: Enhancing Market-Based Testing at Thumbtackmarketplace2023-07
How to Size For Online Experiments With Ratio Metricstravel2023-03
Measuring marketing incremental impacts beyond last click attributionsocial2022-07
What is a p-value? Is one number enough, and where do power and confidence intervals come in?general2022-06
Geo-based AB Testing and Difference-in-Difference Analysis in Instacart Catalogfood-delivery2022-06
Meet Dash-AB: The Statistics Engine of Experimentation at DoorDashfood-delivery2022-05
Balancing Network Effects, Learning Effects, and Power in Experimentsfood-delivery2022-02