Problem
Chips is a video social network. Its recommendations team ranked threads in the feed by popularity, then tried personalized models. On the holdout set the models ranked content like popularity sorting, interpretation showed no personal traits, and an A/B test against popularity showed no significant metric changes. The team had to find out whether the code was broken, users did not care what they watch, or the feed lacked diverse content.
Example from the article: five independently built models with different targets all mimicked popularity sorting, so a shared bug was unlikely, and the cause had to be in the feedback itself.
Solution in brief
Test "users have no preferences" directly: hand-label 60 videos in two niche categories that users should distinguish if they care, show them in an A/B test, and check whether embeddings and ranking metrics become meaningful. They did, so the cause is a shortage of diverse content. To scale, a classifier with uncertainty labels confident cases and sends the rest to humans.
Questions this case answers
- How to tell whether personalized ranking fails because of a pipeline bug or because users give no individual preference signal
- How to bootstrap a recommender for a young platform with little diverse content using a small manual labeling batch to test a hypothesis
- How to find out why personalized recommendations do not beat popularity sorting
- How to tell a bug in the pipeline from the absence of personal preferences in user feedback
Tools and methods
- Popularity sorting — non-personalized baseline.
- ALS, then LightFM — collaborative models; content-only models were tried and did worse.
- Manual labeling — categories: question-and-answer, cars, "girly romance".
- ROC-AUC on category pairs — checks whether embeddings separate labeled categories.
- Classifier ensemble with uncertainty — planned scaling of labeling.
- MAP, precision@k, NDCG — offline ranking metrics on a time split.
Google or a one-time email code. No password.