Experimenting with Machine Learning to Target In-App Messaging

Euan DowersJun 2023

Problem

Spotify shows in-app messages (modals, full-screen takeovers) while a user is in the app. The channel is direct, so a badly timed message can interrupt listening. A randomized holdout that gets no in-app messages shows a mixed effect on user behavior across the whole population. Some users gain from a message, others gain nothing. Sending to everyone therefore annoys some users and wastes the channel.

Example from the article: a user who might enjoy Premium benefits from a subscription prompt, while a user content with the current offering gets no value from a message.

Solution in brief

Predict the causal effect of messaging per user (the conditional average treatment effect, CATE) and message only users with a positive predicted effect. One shared neural network feeds four heads: treatment and control for each of two business metrics. A weighted sum of the two uplifts gives the eligibility score.

Questions this case answers
  • How to choose which units get an intervention when its average effect is mixed and a randomized holdout already exists
  • How to learn one treatment-effect model that balances two competing outcome metrics
  • How to evaluate a targeting policy offline before exposing it to users in an online test
  • How to decide which users should receive an in-app message when the average effect of the message is mixed?

Tools and methods

  • CATE / uplift modeling — predict the per-user effect of a message instead of the outcome itself
  • S-learner and T-learner — the baseline meta-learners the model is derived from
  • Multi-headed shared network (Dragonnet-style) — one model, no treatment feature, separate head per treatment
  • Multi-task learning — two outcome metrics trained together
  • Holdout group — randomized data for both effect estimation and training
  • Offline policy evaluation — compare send-to-all, send-to-none and the model without live traffic
Continue reading — it's free

Google or a one-time email code. No password.