Problem
A classifieds site such as Auto.ru has filters, paid ad promotion and two seller types: individuals and dealers. Few users set many filters, so most browse broad listings, and the order of the top ads decides whether they complete their task. Filters give exact answers, so it is unclear what ML adds. Ranking must also stay fast: scoring about 100 thousand ads per query with a personal model would slow search down.
Example from the article: a user searches a broad listing, and the system should put on top the ads that user is most likely to call about.
Solution in brief
A first-level ad quality model scores the whole ad base at indexing, about every 5 minutes. A second-level personalized CatBoost model (YetiRank loss) re-ranks only the top ads at query time using real-time user factors. Business rules then adjust the order. Models train on impressions, clicks and calls, and are judged by offline ranking metrics and strict A/B tests.
Questions this case answers
- How to add a personalized ranking stage to a marketplace listing when scoring all candidates at query time is too slow
- How to choose a training target and evaluate ranking models when the main business metric is insensitive and easy to inflate
- How to add personalized re-ranking to a listing search that already has many filters and paid promotion?
- How to keep search latency low when a personalized model cannot score every matching ad?
Tools and methods
- CatBoost — trains both ranking levels; YetiRank is the loss for the personalized model
- Pointwise ranking — first-level model: call probability on the day after publication
- Pairwise learning to rank — second-level personalized re-ranking of the top ads
- Negative sampling — random negatives, or proportional to the square root of object popularity
- A/B testing with an MDE table — decides which model stays in production
Google or a one-time email code. No password.