Analyzing Public Pedagogical Benchmarks for AI Tutors and Building Synthetic Students for Testing

Airat Azbukhanov23 May 2026

Problem

Teams building an LLM tutor (Yandex Education: math tutors for grades 5–11 and a "tutor" mode in Alice) find that standard benchmarks such as MMLU, MATH and GSM8K do not measure teaching. They are mostly single-turn and score only "solved / not solved", so they cannot tell giving the answer from leading the student to it. A strong math solver tends to leak answers, and a "be a teacher" system prompt is forgotten after about 8 rounds.

Example from the article: SafeTutors reports about 18% pedagogical harm in single-turn tests and about 78% in multi-turn ones, so a tutor that looks safe turn by turn may fail over a long dialogue.

Solution in brief

Run your tutor on open pedagogical benchmarks (MathTutorBench, MRBench, CDPK) and judge dialogues on your own nine criteria of educational value. To get many multi-turn dialogues for evaluation and fine-tuning, simulate the student with three layers: archetype (weak, average, strong), emotion (a 7×7 Markov chain) and intent plus error type.

Questions this case answers
  • How to evaluate a conversational assistant on teaching quality over many turns rather than on answer correctness of a single reply
  • How to simulate a user with archetypes, emotional states and error types to generate multi-turn dialogues for evaluation and training when labeled dialogues are scarce
  • How to evaluate an AI tutor on pedagogical quality beyond answer correctness
  • How to run open pedagogical benchmarks on your own tutor model in an evening

Tools and methods

  • MathTutorBench — multi-turn, fully open; scaffolding, Socratic questioning and mistakes, with an open reward model
  • MRBench (BEA 2025) — 192 dialogues, gold labels on 8 dimensions; calibrates your LLM-as-judge
  • CDPK / Pedagogy Benchmark — multiple-choice smoke test
  • TutorBench, SafeTutors, EduBench — partially open benchmarks
  • Simulated student — archetype, emotion Markov chain, intent-error table
Continue reading — it's free

Google or a one-time email code. No password.