Method

Bootstrap

Resampling for confidence intervals and variance, incl. block bootstrap, Wilson/Wald intervals.

25 cases from 20 companies apply Bootstrap. 4 were checked in an online A/B test; 6 report results in production.

Typical tasks: Get experiment results faster, Measure impact without an A/B test, Avoid false wins in A/B tests, Evaluate models and 13 more. Companies: Sber, Airbnb, DoorDash, Indeed, Databricks, Expedia Group and 14 more. Often combined with: Power analysis / MDE, t-test, A/B test, A/A test, Bayesian A/B test and 65 more.

Bootstrap Confidence Intervals for LLM Evaluationgeneral2026-07
From Eight Hours to Eight Minutes: Automating A/B Test Analysis on Databricksgaming2026-06
Evolution of the Databricks Commercial Motion: Building Scalable Intelligence on Databrickssaas-b2b2026-06
Double Machine Learning vs Propensity Score Matching: How to Evaluate Non-Randomized Experiments Faster and More Reliablymarketplace2026-05
Problems and Prospects of Causal AIgeneral2026-05
Why Reports Mislead, or How to Measure the Effect of an LLM Featuregeneral2026-05
Diagnosing LLM-Based Ranking Models in Cold-Start Recommender Systems: Coverage, Diversity and Practical Fixesmedia-streaming2026-05
The New Policy Hasn't Run Yet: Comparing It to the Old One Before an A/B Testbanking2026-05
Interleaving for Accelerated Testingtravel2026-02
How We Launched a New Promotion Model: Matching and Regional Testsclassifieds2024-05
So A/B Tests Are Just OLS Regression?general2024-05
Airbnb Brandometer: Powering brand perception measurement on social media data with AItravel2024-04
Milestones on our Journey to Standardize Experimentation at The New York Timesgeneral2024-03
Order Forecasting on a Map Using a Single Time Seriesgeneral2023-10
Accelerating Ranking Experimentation at Thumbtack with Interleavingmarketplace2023-07
Metrics and Loss Functions in Machine Learning System Design—general2023-05
Bootstrapping Time Seriesgeneral2023-05
Improving the customer’s experience via ML-driven payment routingfintech2023-01
Adapted Switch-back Testing to Quantify Incrementality for App Marketplace Search Adsfood-delivery2022-11
Faster ML Experimentation at Etsy with Interleavinge-commerce2022-08
Automation of Strategy Generation for Anomaly Detection in E-Commercee-commerce2022-07
Meet Dash-AB: The Statistics Engine of Experimentation at DoorDashfood-delivery2022-05
Artificial Counterfactual Estimation (ACE): Machine Learning-Based Causal Inference at Airbnbtravel2022-03
Comparing Quantiles at Scale in Online A/B Testingmedia-streaming2022-03
Experimentation is a major focus of Data Science across Netflixmedia-streaming2022-01