Problem
Ranking by engagement alone promotes clickbait: it pulls up content that gets clicks but gives limited long-term engagement. Pinterest wanted its feeds to show content that the average user finds high quality. Nobody could write guidelines for "high quality" without asking users, and rating every item in a corpus of billions is impossible because surveys must not bombard users.
Example from the article: Pinterest asked users to rate image visual appeal on a 1–5 scale and trained a model on 5k rated Pins. The model was added to Homefeed, Related Pins and Search.
Solution in brief
Collect at least 10 survey ratings for each of 5k Pins and take the mean rating as the quality label. Train a small neural network on item embeddings with a pairwise margin ranking loss, comparing only images from the same interest vertical. Use the model score as a ranking feature and validate it in A/B tests.
Questions this case answers
- How to turn a small set of human quality ratings into a per-item quality score that a large-scale ranking system can use
- How to train a scorer on noisy mean ratings when only the relative order of items matters downstream
- How to teach a recommender what users perceive as high-quality content when engagement rewards clickbait?
- How to build a content quality signal from only a few thousand surveyed items?
Tools and methods
- In-app survey — 1–5 rating of single Pins, at least 10 responses per Pin
- Pairwise margin ranking loss with variable margin — learns which of two Pins the average user rates higher
- Fully-connected network (92k parameters) — maps item embeddings to a 0–1 quality score
- NDCG@20 and pairwise ranking accuracy — offline metrics
- Online A/B experiments — check the feature on Homefeed, Related Pins and Search
Google or a one-time email code. No password.