Problem
Search and recommendation teams choose ranking metrics and tests that decide which model ships. A metric such as nDCG assumes how attention falls with position. Engagement-only optimization can miss diversity and long-term value. Classic A/B tests of rankers are slower than interleaving. Offline metrics are useful only if they correlate with online metrics and business KPIs.
Example from the article: ShareChat (about 400 million users) compared the nDCG discount with its own user analytics. After the 100th video in a feed, nDCG gave a much too high probability of being viewed, and a power law fitted real behavior better.
Solution in brief
Treat every ranking metric as a model of user behavior and check it against logs and long-term KPIs. Use a different metric at each pipeline stage, test the position discount against observed views, compare rankers with interleaving, and add diversity and long-term measures next to engagement. The post is a recap, so parameters are few.
Questions this case answers
- How to evaluate and compare rankings with metrics whose assumptions about user attention and long-term value have been checked against real behavior
- How to choose ranking metrics for each stage of a recommendation pipeline?
- How to check that the position-bias discount in nDCG matches what users actually view?
- How to compare two rankers faster than with a classic A/B test?
Tools and methods
- Two-stage recommender — wide candidate generation scored by Recall@100, then a precision stage scored by nDCG@5 (Expedia).
- Position-bias modeling — fit a viewing-probability function to logged behavior instead of assuming nDCG's discount (ShareChat).
- Interleaving — blend results of rankers A and B in one list and read feedback on both at once.
- Result diversification — hierarchical: stance first, topics second.
- Hybrid search — keyword and semantic retrieval together.
Google or a one-time email code. No password.