Faster ML Experimentation at Etsy with Interleaving

Ami Wulf, Ruixi Fan, Adam Henderson, Ercan Yildiz8 Aug 2022

Problem

Etsy's product and ML teams test every new ranking model on live traffic. A standard A/B test of a search ranker needs a large sample, because users differ a lot from each other and that variance hides the effect. Teams with little traffic wait long for an answer, and some models never get a test slot.

Example from the article: Two search algorithms return different orderings of listings for the query "clock". An A/B test sends each user to one algorithm. Interleaving shows every user one merged list built from both.

Solution in brief

Run each search query through both rankers and show the user one list that weaves their results together. Record which ranker supplied each listing, credit a purchase to the ranker that supplied the purchased listing, and aggregate the credits across users. Etsy needed 10X to 100X less traffic than for an equivalent A/B test.

Questions this case answers
  • How to compare two rankers of an ordered result list faster than a user-level randomized test allows, when traffic for testing is scarce
  • How to validate a new experiment method and its infrastructure for bias, false positives and latency before using it for product decisions
  • How to compare two search ranking algorithms with far less traffic than an A/B test needs?
  • How to build an interleaving experiment system on top of an existing search pipeline?

Tools and methods

  • Team-draft interleaving (variation) — builds the merged list
  • Interleaver — layer that splits a request in two and merges the results
  • Apache Spark, Apache Kafka — daily attribution job over search and user-action events
  • A/A and "A-vs-worse-A" tests — validation of bias, false positives and sensitivity
  • Bootstrap — confidence intervals for preference
Continue reading — it's free

Google or a one-time email code. No password.