Improving Instagram notification management with machine learning and causal inference

Nailong Zhang31 Oct 2022

Problem

Instagram sends daily digest push notifications that list stories ready to view. A click-through rate (CTR) model drops notifications with a low predicted click probability. This raised the average click rate, but a large share of digests still went to highly active users. Many of them would view the same stories organically, so the notification adds little and may feel like spam. Cutting sends is risky: if the wrong users lose the notification, they may become less active.

Example from the article: a highly active Instagram user receives a daily digest of stories they would have viewed organically anyway.

Solution in brief

Treat sending as a budget allocation problem and rank notifications by incremental value of sending versus dropping, not by click probability. A randomized experiment drops each eligible notification with 50 percent probability and trains a neural-network uplift model. Only the top share r is sent, and an online quantile transformation keeps the rate at r when scores drift.

Questions this case answers
  • How to allocate a limited volume of interventions to the units with the highest incremental effect rather than the highest response probability
  • How to turn a drifting model score into a stable decision rate when items are scored one at a time online
  • How to decide which push notifications to send when the sending volume has a fixed budget?
  • How to cut notifications to users who would see the content anyway without hurting their activity?

Tools and methods

  • Uplift modeling — a neural network predicts per user the change in activity caused by sending the notification.
  • Randomized send/drop experiment — supplies unbiased training data for the uplift model.
  • Online quantile transformation — maps raw uplift scores to a uniform distribution while preserving order, so a fixed threshold gives a stable send rate.
Continue reading — it's free

Google or a one-time email code. No password.