How Khan Academy Is Building a Better AI Tutor: Our Most Recent Learnings

Khan Academy1 May 2026

Problem

Khan Academy's AI tutor Khanmigo answers students' questions during practice. Any change to it, such as a faster model, a different prompt or more student data, can help one dimension and hurt another. Over six months (October 2025 to April 2026) the team ran about 20 substantive tests on more than 15 million tutoring threads to decide what to ship.

Example from the article: a faster model for the math-checking sub-agent cut response time by 0.3 seconds while math accuracy held steady.

Solution in brief

Each test has one primary metric and a fixed set of guardrails. The metrics are response latency, next-item correctness (did the student answer the next problem on the same skill without help) and an automated rating of cognitive engagement. A change ships when the A/B platform's chance to win exceeds 0.95 and no guardrail moves the wrong way.

Questions this case answers
  • How to run many small A/B tests on an LLM feature so that each change is shipped only if the primary metric improves and no guardrail degrades
  • How to measure the quality of an LLM conversation at scale when the outcome (learning) is delayed and only a proxy is observable
  • How to choose a primary metric and guardrails for A/B tests of an LLM tutor feature?
  • How to make an LLM feature faster without hurting answer quality?

Tools and methods

  • A/B test with a chance-to-win rule — ship if the probability of beating control is above 0.95
  • LLM-as-a-judge — labels each thread passive, active or constructive
  • Guardrail metrics — answer give-aways, math error rate, interactions per thread
  • Context engineering — adds student history to the prompt, one source at a time
Continue reading — it's free

Google or a one-time email code. No password.