Meaningful metrics: How data sharpened the focus of product teams

Erin Gustafson, Ph.D.17 Feb 2023

Problem

After years of strong growth, Duolingo's daily active users (DAU) stalled in 2018. The team that owned DAU could not build A/B tests with a visible effect, because DAU is one lump that many behaviors feed into. Teams need a smaller metric that still matters to the business and is easier to move.

Example from the article: in 2018 Duolingo split DAU into user states, found the rate that moved DAU most, staffed a team on it, and DAU has grown about 4x since 2019.

Solution in brief

Put every user, past or present, into one of 7 mutually exclusive activity states each day and track the daily rates of moving between states (a Markov model). Raise each rate by the same amount in a simulation and see which lifts DAU most. Give that rate to a team and confirm with A/B tests.

Questions this case answers
  • How to decompose a stagnating top-line engagement metric into mutually exclusive user states with transition rates, so that teams can optimize a lower-level metric they can move
  • How to rank candidate lower-level metrics by their downstream effect on the top-line metric before spending experiment capacity
  • How to find metrics a product team can actually move when the top-line daily active users (DAU) metric has stagnated?
  • How to break DAU into lifecycle segments and see which retention or reactivation rate matters most?

Tools and methods

  • Markov model of activity states — splits DAU into 7 states with daily transition rates
  • Growth simulation — raises each rate in turn and measures the effect on DAU
  • A/B tests on the chosen rate — check that the rate is movable and that it moves DAU
  • User-identity resolution (aliasing) — keeps one person from appearing as a new user on a new device
Continue reading — it's free

Google or a one-time email code. No password.