How The New York Times Uses Machine Learning To Make Its Paywall Smarter

Rohit Supekar10 Aug 2022

Problem

The New York Times lets non-subscribers read a fixed number of free articles per month (the meter limit). Originally the limit was the same for everyone. A tighter limit gives more subscriptions but cuts readership, and a looser one does the opposite, so the limit for each registered user has to balance the two. The catch: nobody sees what the same user would have done under another limit.

Example from the article: In a randomized control trial (R.C.T.) with different meter limits for registered users, a higher limit raised average page views and lowered the subscription conversion rate.

Solution in brief

The Dynamic Meter, a causal model, trains two S-learner-style models on R.C.T. data, one for subscription propensity and one for engagement. A weight δ blends them into one score, and each user gets the meter limit with the best score. Sweeping δ gives a Pareto front, which is evaluated offline by inverse probability weighting on R.C.T. data.

Questions this case answers
  • How to assign each user one of a few discrete treatments so that two conflicting outcomes are traded off explicitly, when the only unbiased training data come from a randomized experiment
  • How to backtest a treatment-assignment policy on historical randomized data when only users whose random assignment matches the policy's choice are observable under that policy
  • How to choose a personalized free-article limit for each registered user instead of one limit for everyone?
  • How to balance subscription conversion against reader engagement when tighter limits raise one and cut the other?

Tools and methods

  • S-learner base models — predict subscription propensity and normalized engagement from user features plus the meter limit as a categorical treatment
  • Pareto front over weight δ — blends the two predictions into one objective; sweeping δ from 0 to 1 gives the set of solutions
  • Inverse probability weighting with Hájek estimation — backtests the policy on past R.C.T. data
Continue reading — it's free

Google or a one-time email code. No password.