Problem
Avito's Monetization Efficiency team keeps changing the auction that ranks paid listings: new auction types, bidding algorithms, CTR models. Each change must be checked on two sides: users must not get worse results, and paid listings must not do worse. Standard A/B setups cannot measure both. Splitting by user does not guarantee that a listing appears in both groups. Splitting by seller never puts one listing under both algorithms. Regional and category tests are noisy and often repeated.
Example from the article: a seller buys a boost for 5 days with a fixed budget. The team suspects that one bidding algorithm spends that budget faster than planned, so the seller loses clicks late in the period. A plain test cannot isolate this.
Solution in brief
Make the promotion campaign the unit of splitting. Each paid listing gets two subcampaigns, one per algorithm, the budget is divided between them, and users are split in the same proportion. Both user and seller metrics can then be compared on identical listings.
Questions this case answers
- How to A/B test a new auction or bidding rule when test groups share one budget and compete for the same paid items, so that both user and advertiser outcomes are measured
- How to randomize at a level where every unit sees both algorithms without the two interfering with each other
- How to A/B test a new auction or bidding algorithm when the budget is shared between test groups
- How to measure the effect on users and on advertisers in one experiment
Tools and methods
- Split test — A/B test where one promotion campaign is divided into subcampaigns with different algorithms.
- User-level A/B split in the budget proportion — X% of users see algorithm 1, (100−X)% see algorithm 2.
- Auction ranking by expected revenue per impression — candidates sorted by bid times predicted CTR.
- Automatic bidding algorithm and CTR model — the two components whose changes are tested.
Google or a one-time email code. No password.