Method

A/B test

Randomized controlled online experiment comparing variants, incl. split, A/B/C and regional tests.

137 cases from 72 companies apply A/B test. 77 were checked in an online A/B test; 26 report results in production. Includes variants: A/B/n test, Bayesian A/B test.

Typical tasks: Avoid false wins in A/B tests, Get experiment results faster, Rank search results, Retrieve and rank from millions of items and 23 more. Companies: DoorDash, Spotify, Pinterest, Avito, Netflix, LinkedIn and 66 more. Often combined with: Gradient boosting, Power analysis / MDE, A/B/n test, Bayesian A/B test, Bootstrap and 188 more.

Measuring the Impact of Personalized Recommendationsmedia-streaming2026-10
Powering AI-led research through simulationfood-delivery2026-10
The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scalemedia-streaming2026-09
Why Spotify Is Not Using Bayesian A/B Testingmedia-streaming2026-09
Correlation Lied to Us: Rethinking Product Impact with Causal Inferenceclassifieds2026-08
GenPage: Towards End-to-End Generative Homepage Construction at Netflixmedia-streaming2026-07
Databricks on Databricks: Build Your Own Experimentation Platformsaas-b2b2026-06
From Eight Hours to Eight Minutes: Automating A/B Test Analysis on Databricksgaming2026-06
Semantic Search for AI Agents at Scale: Retrieval and Ranking for LinkedIn’s Hiring Assistantsocial2026-06
TargetRun: AI-Powered Trip Prediction for Delivering Timely Offersretail2026-06
AI Blender in Search: How We Stopped Worshipping Clicks and Moved Block Placement to LLMssearch-engine2026-05
ML in Store Operations and Logistics: How Production Models Directly Optimize Processesretail2026-05
A Reranker in Retail: Turning Advertising into Recommendationsretail2026-05
How to Integrate AI into Products When They Are All Differentsaas-b2b2026-05
ABsalute: A/B Testing Automationretail2026-05
Why Reports Mislead, or How to Measure the Effect of an LLM Featuregeneral2026-05
How Khan Academy Is Building a Better AI Tutor: Our Most Recent Learningsedtech2026-05
Better Experiments with LLM Evals — A funnel, not a forkmedia-streaming2026-05
We Tested Marketing Incentives to AI Agents. Here's What Happened.fintech2026-04
How Vinted Serves Personalised Search Autocompletemarketplace2026-04
The secret sauce to growing search KPIs: A/B testinge-commerce2026-04
Interleaving for Accelerated Testingtravel2026-02
How Yelp Built a Back-Testing Engine for Safer, Smarter Ad Budget Allocationadtech2026-02
Adapting the Facebook Reels RecSys AI Model Based on User Feedbacksocial2026-01
Building AI search: from prototype to productionsearch-engine2026-01
Why we use separate tech stacks for personalization and experimentationmedia-streaming2026-01
Improving Quality of Recommended Content through Pinner Surveyssocial2025-12
Normalized Entropy or Apply Rate? Evaluation Metrics for Online Modeling Experimentsmarketplace2025-11
Mercari's Global Search Backend Design and Search Platform Expansionmarketplace2025-10
Unlocking Faster Insights with Experimenter-Defined Segmentationse-commerce2025-10
Scaling Subscriptions at The New York Times with Real-Time Causal Machine Learningmedia-streaming2025-10
How DoorDash Ads keep consumers first with budget A/B experimentationadtech2025-09
The New User Dilemma: Why We Killed Our Heuristics (& What We Built Instead)marketplace2025-09
Beyond Winning: Spotify's Experiments with Learning Frameworkmedia-streaming2025-09
Comment ranker - An ML-based classifier to improve LLM code review quality using Atlassian's proprietary datasaas-b2b2025-08
A scalable LLM approach to enhancing chatbot knowledge with user-generated contentfood-delivery2025-08
Advancing Menu Content with AI: How DoorDash uses AI to generate menu descriptionsfood-delivery2025-06
Measuring Commercial Impact at Scale at Canvasaas-b2b2025-06
How the Effect Forecast for Paid Listing Promotion at Avito Works: Algorithms and Metrics Under the Hoodclassifieds2025-05
How LLM Automates Work at Avitoclassifieds2025-05
Description Generation One Year Later. What We Managed to Understand Over the Yearclassifieds2025-05
Ranking VK Storiessocial2025-05
How We Saved Couriers' Time with Shift Recommendations in Lavkafood-delivery2025-05
Runtime Transformer in Personal Recommendations at Yandex Lavkafood-delivery2025-05
The Social Graph as an Ad Targeting Tooltelecom2025-05
Cooking the Treats Right: How ML Helps Run Promotions Efficientlybanking2025-05
Renting Out Smartly: Launching a DS Product in Short-Term Rentalsclassifieds2025-05
Validating Propensity Score Matching: Metrics and Testsmobility2025-05
Optimizing Regular Prices: Combining Experimental Approaches, ML and Economic Theoryretail2025-05
500X Scalability of Experiment Metric Computing with Unified Dynamic Frameworksocial2025-05