Multi-Relevance Ranking Model for Similar Item Recommendation

Chen Xue, Rachel Vishnepolsky, Zhen Ge14 Jun 2022

Problem

eBay's Promoted Listings Standard similar-item engine shows sponsored listings on the View Item page, ranked by purchase probability. The ranker was trained on binary labels: an item was relevant only if it was bought with the seed item. Users usually buy one item from a list, so other good recommendations were labeled negative. Purchases are also far rarer than clicks, so even with oversampling the model had little positive data. Weaker signals such as clicks never reached the model.

Illustrative example: a buyer clicks two recommended items, watchlists one and buys neither. Both count as irrelevant, as does every item that was never clicked.

Solution in brief

Replace the binary label with an ordered relevance scale: no selection, selection, the four buyer actions (watchlist, cart, offer, Buy It Now), then purchase on top. The gradient boosted tree takes the ordinal labels, and tuned per-label sample weights in the pairwise loss set how much more each level counts. The click-level version raised purchases by 2.97% in an A/B test.

Questions this case answers
  • How to train a ranker for a rare conversion event when many weaker engagement signals exist but binary labels ignore them
  • How to encode graded feedback levels in a pairwise-loss tree model that supports ordinal labels but not their magnitudes
  • How to train a similar-item ranker when purchases are too rare and a binary purchase label discards useful signal?
  • How to turn clicks, watchlist adds, cart adds and offers into graded relevance labels?

Tools and methods

  • Multi-relevance labels — graded relevance from engagement actions instead of purchase / no purchase
  • Gradient boosted tree ranker, pairwise loss — ranks recommended items per seed item by purchase probability
  • Sample weights per label — set, by hyperparameter search, how much more relevant one level is than another
  • Offline evaluation and A/B test — purchased-item rank offline, purchases and revenue online
Continue reading — it's free

Google or a one-time email code. No password.