Problem
Duolingo shows non-subscribers one ad after each learning session: an in-house ad promoting a subscription, or a network ad that earns money from an outside advertiser. By 2023 the choice was driven by years of rules added after A/B tests. The rules were spread over different parts of the codebase, hard to reason about, and prone to bugs. Leadership doubted an ML system would pay for itself, and the first model would cover only one screen.
Example from the article: the old logic was a tree of checks on learner properties, such as which session of the day the learner was on, ending in either ad type.
Solution in brief
Replace the rule tree with one model that decides the ad type. It was built with tools already in place: dbt pipelines for fresh data and BigQuery ML for training and inference in SQL. After a first model that predicted purchase probability, the team switched to modeling causal uplift of the in-house ad, using XGBoost as a contextual bandit.
Questions this case answers
- How to replace accumulated hand-written decision rules with a model that maximizes incremental revenue per user
- How to prove the value of an ML decision system quickly, using only the existing data stack
- How to retrain a decision model on logs it generated itself without drift between training and serving populations
- How to replace a tangle of hand-written rules that choose between an in-house promo and a network ad with a single model?
Tools and methods
- dbt with dbt_ml — runs the recurring data job and BigQuery ML training from the existing pipelines
- BigQuery ML — trains, tunes and evaluates off-the-shelf models; inference is a SQL query
- XGBoost — the model, turned into a contextual bandit on uplift
- Random serving — occasionally ignoring the model and showing an ad at random, in place of a holdout group
Google or a one-time email code. No password.