Task

How to avoid false wins in A/B tests

Keep A/B results honest with many metrics, variants and peeks: multiple-testing control, SRM checks, Bayesian readouts, novelty effects.

Why Spotify Is Not Using Bayesian A/B Testingmedia-streaming2026-09
From Eight Hours to Eight Minutes: Automating A/B Test Analysis on Databricksgaming2026-06
ABsalute: A/B Testing Automationretail2026-05
Why Reports Mislead, or How to Measure the Effect of an LLM Featuregeneral2026-05
The New Policy Hasn't Run Yet: Comparing It to the Old One Before an A/B Testbanking2026-05
How Khan Academy Is Building a Better AI Tutor: Our Most Recent Learningsedtech2026-05
Measure Less to Learn More: Using Fewer, Higher-quality Metrics to Capture What Matterssocial2026-04
The secret sauce to growing search KPIs: A/B testinge-commerce2026-04
Why we use separate tech stacks for personalization and experimentationmedia-streaming2026-01
Accelerating experimentation at DoorDash with a multi-armed bandit platformfood-delivery2025-12
LLM-Powered Relevance Assessment for Pinterest Searchsocial2025-12
Reducing experiment duration with predicted control variatese-commerce2025-11
Efficient Optimization With Ax, an Open Platform for Adaptive Experimentationsocial2025-11
Speed with Rigor: testing smarter with group sequential designclassifieds2025-08
Measuring Commercial Impact at Scale at Canvasaas-b2b2025-06
Optimizing Regular Prices: Combining Experimental Approaches, ML and Economic Theoryretail2025-05
Improve Your Next Experiment by Learning Better Proxy Metrics From Past Experimentsmedia-streaming2024-08
Safeguarding app health and consumer experience with metric-aware rolloutsfood-delivery2024-07
How we build experiments in-housesaas-b2b2024-06
Make Better Decisions by Embracing Uncertaintyfintech2024-06
So A/B Tests Are Just OLS Regression?general2024-05
Fixed-Power Designs: It’s Not IF You Peek, It’s WHAT You Peek atmedia-streaming2024-05
WISE: Skyscanner’s Bayesian AB experimentation library and decision enginetravel2024-04
Grab Experiment Decision Engine - a Unified Toolkit for Experimentationmarketplace2024-04
Sequential A/B Testing Keeps the World Streaming Netflix. Part 1: Continuous Datamedia-streaming2024-03
Sequential A/B Testing Keeps the World Streaming Netflix Part 2: Counting Processesmedia-streaming2024-03
Risk-Aware Product Decisions in A/B Tests with Multiple Metricsmedia-streaming2024-03
Experiment Faster and with Less Effortfood-delivery2024-02
Addressing the Challenges of Sample Ratio Mismatch in A/B Testingfood-delivery2023-10
Lessons Learned From Running Web Experimentsfintech2023-07
Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0media-streaming2023-07
Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testingmedia-streaming2023-07
Bayes in the Wildgeneral2023-05
A/B tests in KIONmedia-streaming2023-05
A Bayesian Approach to A/B Tests: A Conversion Test Examplegeneral2023-05
Using Metrics Layer to Standardize and Scale Experimentation at DoorDashfood-delivery2023-04
The Data Infra Behind Zillow's 3x Growth in Experiment Volumemarketplace2023-03
Choosing a Sequential Testing Framework — Comparisons and Discussionsmedia-streaming2023-03
Mitigating the winner's curse in online experimentse-commerce2022-12
Understanding the collective impact of experimentse-commerce2022-12
Balancing Velocity and Confidence in Experimentationfood-delivery2022-11
Beyond A/B test : Speeding up Airbnb Search Ranking Experimentation through Interleavingtravel2022-10
How Bandit Testing Improves A/B Tests at Wealthfrontfintech2022-06
What is a p-value? Is one number enough, and where do power and confidence intervals come in?general2022-06
Meet Dash-AB: The Statistics Engine of Experimentation at DoorDashfood-delivery2022-05
Challenges in Experimentationmobility2022-04
Imbalance detection for healthier experimentationmarketplace2022-03
Search journey towards better experimentation practicesmedia-streaming2022-02
Experimentation is a major focus of Data Science across Netflixmedia-streaming2022-01