Problem
Recommenders are usually judged by relevance and ranking metrics such as NDCG and precision, which do not reward variety. A price-sensitive traveler who sees five properties at nearly the same price has nothing to compare and may stop exploring. Expedia's recommendations team wanted more variety in its lists without retraining every model and without a large loss in relevance. That needs a metric for variety and a way to reorder existing model output.
Example from the article: five hotels priced at $150, $140, $155, $135 and $147 shown to a price-sensitive traveler, who may become unhappy and unwilling to explore further.
Solution in brief
Measure diversity as intra-list dissimilarity: the average pairwise distance between items in the top-k list. Raise it by reranking model output with Maximal Marginal Relevance (MMR, called the Neighborhood Cosine Distance approach in the article), where one weight balances relevance against distance. Use explicit item features rather than trained embeddings, and pick the weight from the offline diversity versus NDCG curve.
Questions this case answers
- How to increase the variety of a ranked list as a post-processing step while keeping the loss in ranking quality under control
- How to quantify the diversity of a ranked list of items so that different models can be compared on it
- How to measure how diverse a recommended list is?
- How to make a recommended list more diverse without retraining the ranking model?
Tools and methods
- Intra-list dissimilarity — diversity metric, usable with any model that has item vectors
- MMR reranking (Neighborhood Cosine Distance) — post-processing with a diversity weight ; the approach Expedia chose
- K-means clustering on explicit features — alternative reranking: take items from different clusters
- Hotel2Vec embeddings — trained property vectors, rejected as the diversity space
- SciPy
cdist, NumPy — pairwise distances for the metric
Google or a one-time email code. No password.