Khan Academy Improves State Test Scores: Results from New 3-Year Efficacy Study

Bogdan Yamkovenko1 Aug 2025

Problem

Product teams want to know whether using a feature improves an outcome. Comparing heavy users with light users does not answer it, because heavy users are usually the more motivated or higher-performing ones. Khan Academy faced this in a district of 13 Newark schools, where about 5200 students were observed over several years. No experiment was available, and the goal was to isolate the effect of Khan Academy practice on state math test scores.

Example from the article: students who simply practiced more looked better, so the team compared each student only to their own earlier years and asked whether a change in practice moved the same student's test score.

Solution in brief

Build a three-year student panel and fit a fixed-effects regression with student, teacher, grade and year effects. Only within-student changes in usage count, so stable traits such as motivation and teacher style cancel out. Lag and lead usage terms check carry-over and reverse causality.

Questions this case answers
  • How to estimate the dose-response effect of repeated product use on an outcome from yearly panel data when users self-select into heavy use and no experiment is possible
  • How to remove stable unobserved user and group traits from an observational effect estimate and test the remaining assumptions
  • How to estimate the effect of product usage on outcomes from multi-year panel data when users cannot be randomized?
  • How to compare heavy and light users without being fooled by motivation and teacher quality?

Tools and methods

  • Panel fixed-effects regression (student, teacher, grade, year) — removes stable confounders by comparing each student with themselves
  • Skills leveled up to proficient or higher (STP+) and learning hours — two usage measures (dose) for the same model
  • Lagged and lead dosage terms — carry-over test and placebo test on the exogeneity assumption
  • Interaction with baseline score — checks whether weaker students benefit more
Continue reading — it's free

Google or a one-time email code. No password.