Method

Statistical hypothesis tests

t-test, Mann-Whitney, chi-square, z-test, KS and permutation tests.

35 cases from 27 companies apply Statistical hypothesis tests. 6 were checked in an online A/B test; 10 report results in production. Includes variants: t-test.

Typical tasks: Get experiment results faster, Avoid false wins in A/B tests, Measure impact without an A/B test, Experiment when users affect each other and 11 more. Companies: DoorDash, Grab, Expedia Group, Spotify, Sber, Yandex and 21 more. Often combined with: t-test, Bootstrap, Power analysis / MDE, A/B test, Interleaving and 67 more.

From Eight Hours to Eight Minutes: Automating A/B Test Analysis on Databricksgaming2026-06
Double Machine Learning vs Propensity Score Matching: How to Evaluate Non-Randomized Experiments Faster and More Reliablymarketplace2026-05
Why Reports Mislead, or How to Measure the Effect of an LLM Featuregeneral2026-05
How to Safely Roll Out New Versions of Product AI Agents Using an Auto-Metrics Systeme-commerce2026-05
Interleaving for Accelerated Testingtravel2026-02
Extracting trending keywords from OpenChat messagessocial2025-09
Causal Methods for Feature Selectiongeneral2025-05
The Quest to Understand Metric Movementssocial2025-02
Introducing the new A/B testing estimatorgeneral2025-01
How DoorDash is pushing experimentation boundaries with interleaving designsfood-delivery2024-08
How We Launched a New Promotion Model: Matching and Regional Testsclassifieds2024-05
How we evaluated the business impact of marketing campaignsmobility2024-05
Grab Experiment Decision Engine - a Unified Toolkit for Experimentationmarketplace2024-04
Milestones on our Journey to Standardize Experimentation at The New York Timesgeneral2024-03
Risk-Aware Product Decisions in A/B Tests with Multiple Metricsmedia-streaming2024-03
Privacy Sandbox Market Testing: a statistical model for measuring performance impactsadtech2023-12
Addressing the Challenges of Sample Ratio Mismatch in A/B Testingfood-delivery2023-10
How to Accurately Test Significance with Difference in Difference Modelsmedia-streaming2023-09
Accelerating Ranking Experimentation at Thumbtack with Interleavingmarketplace2023-07
A Bayesian Approach to A/B Tests: A Conversion Test Examplegeneral2023-05
Bootstrapping Time Seriesgeneral2023-05
ML System Design Doc Challenge – Finalbanking2023-05
Market Segmentation for Geo-Testing at Scaletravel2023-02
Increase A/B Testing Power by Combining Experimentse-commerce2022-10
Beyond A/B test : Speeding up Airbnb Search Ranking Experimentation through Interleavingtravel2022-10
Optimizing Markdown in Fashion E-Commerce with Machine Learninge-commerce2022-08
Feature Investigation: Automatically Detect Drift in your Data Over Timefintech2022-08
Measuring downstream impact on social networks by using an attribution frameworksocial2022-07
Study of the effectiveness of various clustering methods for analyzing the structure of gamma-hadron familiesgeneral2022-06
What is a p-value? Is one number enough, and where do power and confidence intervals come in?general2022-06
Automated Experiment Analysis - Making experimental analysis scalablemobility2022-05
Meet Dash-AB: The Statistics Engine of Experimentation at DoorDashfood-delivery2022-05
Using Gamma Distribution to Improve Long-Tail Event Predictionsfood-delivery2022-04
Using Machine Learning to Understand How Branding in Photos Affects the Car Shopping Experiencemarketplace2022-03
Imbalance detection for healthier experimentationmarketplace2022-03