The Field Guide to Non-Engagement Signals

Leif Sigerson, Wendy Matheny27 Mar 2024

Problem

A feed ranked only on predicted engagement can surface content people click but do not value: clickbait, low-quality or harmful items. Platforms struggle to add other signals for two reasons. Non-engagement signals come from surveys or expert labels, so they exist at a tiny fraction of the scale of logged clicks. Their payoff is slow: they can lower short-term engagement (for example by removing clickbait) and help retention only later. Small and mid-size companies rarely have the budget for experiments long enough to see it.

Example from the article: Pinterest's CEO describes ranking that surfaces "the car crash we can't look away from", which people would decline to see again if asked afterwards.

Solution in brief

Pinterest and partners collected 140+ public studies and tested them in a workshop with practitioners from 7 platforms. The 19 propositions that held up form the Field Guide. Use it as evidence where you lack your own analysis. Add quality labels and item-level surveys to ranking, aim at specific well-being drivers, and judge by long-term retention.

Questions this case answers
  • How to add sparse, human-labeled quality signals to a ranking model trained on abundant behavioral signals when their benefit shows only in long-term retention
  • How to choose which published findings to rely on when internal evidence on a ranking decision is weak or missing
  • How to stop a feed ranked on predicted engagement from surfacing clickbait and low-quality content?
  • How to add survey answers and content-quality labels to ranking and judge them by long-term retention?

Tools and methods

  • Item-level in-app surveys — users say directly what they want to see
  • Expert quality labeling — independent human assessment of content quality, used as a ranking weight
  • Generative AI labeling — a cheaper way to scale quality signals, with humans still in the loop
Continue reading — it's free

Google or a one-time email code. No password.