Problem
Yelp splits each ad campaign's spend between its own inventory (website, mobile site, app) and the off-platform Yelp Ad Network. The split must meet advertisers' goals and grow revenue. Each day's budget depends on the outcomes of previous days, so a small algorithm change can cascade through the whole billing period.
A/B tests fit this setting badly. Experiments run at the advertiser level, so samples are small. Most advertisers set monthly budgets, so a readout takes a month. A mistake hits real advertisers.
Example from the article: testing a new allocation method ("Algorithm X") together with a tuning constant (parameter Alpha) on December 2025 campaigns.
Solution in brief
Replay the whole allocation process on historical campaign data. Production budgeting and billing code runs unchanged, and ML models replace the outcomes the system cannot control. An optimizer searches parameter values, and only the best candidates go to a live A/B test, which stays the final validation.
Questions this case answers
- How to preview the effect of a change to a sequential allocation algorithm on historical data when each day's decision depends on previous outcomes and a live test takes a month
- How to estimate counterfactual outcomes of a different spend level when the outcome depends on external systems and random user behavior
- How to test a change to a budget allocation algorithm before a monthly A/B test?
- How to simulate a feedback-driven system day by day on historical data?
Tools and methods
- Back-testing engine — daily replay of each campaign with candidate parameters
- Scikit-Optimize (Bayesian search) — proposes the next candidate to simulate
- CatBoost — predicts impressions, clicks and leads from daily budget and campaign features
- Poisson sampling — turns expected values into integer outcomes
- MLflow — logs parameters and metrics of every candidate
- Git submodules — pull production repositories into the engine
Google or a one-time email code. No password.