How The New York Times Incorporates Editorial Judgement in Algorithms to Curate Home Screen Content

Zhen Yang2 Oct 2024

Problem

The New York Times publishes over 250 stories a day, but its home page shows only 50 to 60 at a time. Editors used to choose and place stories by hand several times a day. That made it hard to keep the page fresh and relevant for each reader, to scale curation, and to expose more stories. A purely algorithmic feed worried editors. It could ignore important stories, bury unpopular ones and leave editors blind to what readers see.

Example from the article: the Features Module, a highly visible block on the web home page that carries features and columns. It moved to algorithmic programming permanently in spring 2024.

Solution in brief

Every algorithmic module runs three steps: pool, rank, finish. Editors define the pool and an importance signal, a contextual bandit ranks within the pool, and rules applied after ranking keep editorial intent in the output. Preview and monitoring tools for editors made the newsroom comfortable with the system.

Questions this case answers
  • How to rank a perishable catalog with a learning algorithm while keeping human priorities and exposure guarantees for items the model would bury
  • How to avoid both stale and constantly churning recommendation slots when users visit at very different frequencies
  • How to let an algorithm fill a home-page module without editors losing control over what readers see?
  • How to combine a contextual bandit with editorial importance scores?

Tools and methods

  • Pool-rank-finish pipeline — structure of every module, with editorial input at each step
  • Contextual bandit — ranks the pool using reading history and location
  • Editorial importance — editor-assigned rank or importance inferred from past promotion
  • Pinning, read and seen filters, Exposure Boosting, Smart Refreshing, Exposure Minimums — post-ranking rules
  • A/B testing — tunes rule parameters with editors
Continue reading — it's free

Google or a one-time email code. No password.