How We Launched a New Promotion Model: Matching and Regional Tests

Alina Savelieva25 May 2024

Problem

Avito's monetization analysts had to test a new way for short-term rental sellers to buy promotion. Sellers share one feed, so a seller-level A/B test leaks: test sellers promote more and take liquidity from control sellers, who then buy less of the old promotion. Revenue is distorted. Splitting by region avoids the leak, but regions are not randomly assigned, and the usual method of picking control regions by pre-period dynamics lacked history and suffered from regional seasonality.

Example from the article: promotion moved from prepaid packages to a seller-chosen commission on each booking. Test sellers sat in test regions, and the control had to be built from all other regions.

Solution in brief

Test by region to cut interference, then match each test seller to the nearest seller from other regions on pre-test revenue, bookings and listing count. Estimate the effect with a t-test with cluster-robust errors. Validate by balance metrics and by repeated A/A simulation on the pre-period.

Questions this case answers
  • How to test a product change when units share one supply pool and unit-level randomization would leak between test and control groups
  • How to build a control group by matching when assignment was not random, and check the matching with simulated null tests on pre-period data
  • How to test a marketplace product change when sellers share one feed and a seller-level A/B test would leak between groups
  • How to build a control group by matching when assignment to the test was not random

Tools and methods

  • Regional (or category) test — removes network effects when liquidity stays inside a region
  • Matching with replacement, Mahalanobis distance — builds the control group
  • FAISS — approximate nearest neighbours, fast enough to repeat matching
  • Cluster-robust t-test — handles repeated control users
  • SMD, PSI, Kolmogorov-Smirnov test — balance after matching
  • Pre-period A/A simulation, bootstrap, Wilson interval — false positive check
Continue reading — it's free

Google or a one-time email code. No password.