Why Khan Academy will be using \"skills to proficient\" to measure learning outcomes (and you should too!)

Bogdan Yamkovenko25 Sep 2023

Problem

Time on platform alone does not show whether learners learn. Khan Academy needed an internal metric that tracks a long-term outcome (math achievement) and that a team can move and explain. The outcome itself is measured by an external standardized test. The author argues that edtech success must be measured by learning, not only by time spent.

Example from the article: Khan Academy compared its progress indicators with MAP Growth mathematics scores and chose the proportion of math skills completed at proficient or mastered level.

Solution in brief

Pick the in-product progress indicator that correlates with the external test score, and check that the relationship holds across volume of work, time spent and grade. At Khan Academy this is "skills to proficient+": the number of skills a learner leveled up to proficient or mastered.

Questions this case answers
  • How to pick an internal progress metric for a product whose real outcome is measured only by an external test, and validate it against that test
  • How to decide between breadth and depth of usage when engagement time alone does not predict the outcome
  • How to choose a product metric that reflects learning, not just time spent on the platform?
  • How to check that an internal progress metric is trustworthy by comparing it with an external standardized test?

Tools and methods

  • Skills to proficient+ — count of skills a learner leveled up to proficient or mastered, used as the internal learning metric
  • Proportion of skills at proficient/mastered — share of practiced skills at the top levels, the form that was validated against the external test
  • MAP Growth mathematics assessment — nationally normed standardized test used as the external validity check
  • Convergent validity check — comparing a candidate metric with an established measure of the same construct
Continue reading — it's free

Google or a one-time email code. No password.