How Duolingo sped up data analysis by 68X with sampling

Jonathan Burket23 May 2025

Problem

Analysts at Duolingo explore event data interactively and chain follow-up queries, so each query should ideally finish in under 10 seconds. The warehouse holds roughly 13 trillion rows and the volume nearly doubles every year. Queries slowed to several minutes. More BigQuery capacity would have meant doubling the spend every year.

Example from the article: "How many learners completed a French lesson yesterday?" followed by "Was there a particular country that saw a spike?"

Solution in brief

Exploratory work needs trends and segment comparisons, not exact counts. So Duolingo queries a fixed subset of users. Each row gets a "sampling value" from a hash of the user ID, and the table is clustered (sorted) by it. A filter on that value then reads only the matching slice of the table.

Questions this case answers
  • How to keep exploratory queries over a fast-growing event table interactive and cheap when full scans take minutes
  • How to draw a repeatable per-user sample inside a warehouse so that a filter reads only a fraction of the data
  • How to keep interactive exploratory queries under about 10 seconds when event tables double in size every year?
  • How to cut warehouse query cost on huge event tables without paying for more compute?

Tools and methods

  • BigQuery clustering — sorts the table by the sampling value so a filter skips most of the data
  • Hash of user_id mapped to 0–1 — a stable per-user "sampling value" stored with every row
  • Per-user sampling — keeps all events of the chosen users
  • Automatic sampling-rate recommendation — the internal analysis tools suggest a rate based on the data volume
Continue reading — it's free

Google or a one-time email code. No password.