Problem
On a marketplace, sellers pay per click, per call or per transaction. Ranking has to account for the expected revenue of each listing, so the team needs models that turn user actions into money. Three choices decide whether the estimates are usable: the target and its attribution to clicks, the weight of clicks from top and bottom positions, and whether scores are calibrated enough to multiply by a bid. The unit's goal is revenue growth without hurting user quality.
Example from the article: a user searches for "bicycle", then "mountain bike", then "mountain bike trek", clicks several listings and finally writes to the seller of listing A. It is not clear which click should get the contact as its target.
Solution in brief
Expected revenue is the conversion probability times the payment (bid, commission or booking cost). The probabilities come from separate CTR and CVR models, checked with stratified AUC for ordering and RIG for calibration. Position bias is handled either by a CatBoost score difference used as a baseline or by inverse propensity weighting of clicks. Clicks from exploratory result pages get a lower weight.
Questions this case answers
- How to estimate expected payment per item for ranking when payment is per click, call or transaction and the probabilities must be calibrated enough to multiply by a price
- How to remove position bias from click logs and attribute a conversion to the right clicks in a multi-step session
- How to estimate expected revenue per listing for search and recommendation ranking when sellers pay per click, call or transaction
- How to calibrate a CTR or CVR model so its scores can be multiplied by a bid
Tools and methods
- CTR model — impression to click, for pay-per-click sellers; CatBoost.
- CVR model — click to contact or target action.
- Stratified AUC and RIG — ranking quality inside queries and calibration quality.
- QueryCrossEntropyLoss — loss for the bid adjustment model.
- Inverse propensity scoring (IPS) — click reweighting for position bias.
Google or a one-time email code. No password.