Problem
Duolingo's product goal is learning, but its Efficacy Lab studies, which measure learning directly, cannot be run for the daily A/B tests the product team needs. Teams need a metric that moves within days and still points toward real learning. The first two candidates fell short. Total sessions favored short, easy sessions over harder new content, and total time was dominated by a small group of studious users.
Example from the article: a learner with 15 minutes a day may do 5 easy sessions or 3 harder ones. Fewer sessions is not worse for that learner, but a session count scores it as worse.
Solution in brief
Duolingo moved from total sessions to time spent learning, then to Time Spent Learning Well (TSLW). TSLW counts minutes, gives full weight to lessons that move the learner down the course path and half weight to everything else, and is tuned so that more learners reach at least 15 minutes a day. Features are then tested against it.
Questions this case answers
- How to define a daily proxy metric for a slow, hard-to-measure outcome so that every experiment can be judged against it
- How to build a time-based engagement metric that resists outliers and gaming and weights valuable activity higher
- How to build a daily-measurable proxy for learning when efficacy studies are too slow for A/B tests?
- How to define an engagement metric that is not dominated by a few heavy users or by session length?
Tools and methods
- Time Spent Learning Well (TSLW) — weighted minutes in learning activities, used as the daily proxy for learning.
- A/B tests — each feature (quests, leaderboard XP, lesson length, new content) is judged on TSLW.
Google or a one-time email code. No password.