Problem
Teams building an LLM tutor (Yandex Education: math tutors for grades 5–11 and a "tutor" mode in Alice) find that standard benchmarks such as MMLU, MATH and GSM8K do not measure teaching. They are mostly single-turn and score only "solved / not solved", so they cannot tell giving the answer from leading the student to it. A strong math solver tends to leak answers, and a "be a teacher" system prompt is forgotten after about 8 rounds.
Example from the article: SafeTutors reports about 18% pedagogical harm in single-turn tests and about 78% in multi-turn ones, so a tutor that looks safe turn by turn may fail over a long dialogue.
Solution in brief
Run your tutor on open pedagogical benchmarks (MathTutorBench, MRBench, CDPK) and judge dialogues on your own nine criteria of educational value. To get many multi-turn dialogues for evaluation and fine-tuning, simulate the student with three layers: archetype (weak, average, strong), emotion (a 7×7 Markov chain) and intent plus error type.
Questions this case answers
- How to evaluate a conversational assistant on teaching quality over many turns rather than on answer correctness of a single reply
- How to simulate a user with archetypes, emotional states and error types to generate multi-turn dialogues for evaluation and training when labeled dialogues are scarce
- How to evaluate an AI tutor on pedagogical quality beyond answer correctness
- How to run open pedagogical benchmarks on your own tutor model in an evening
Tools and methods
- MathTutorBench — multi-turn, fully open; scaffolding, Socratic questioning and mistakes, with an open reward model
- MRBench (BEA 2025) — 192 dialogues, gold labels on 8 dimensions; calibrates your LLM-as-judge
- CDPK / Pedagogy Benchmark — multiple-choice smoke test
- TutorBench, SafeTutors, EduBench — partially open benchmarks
- Simulated student — archetype, emotion Markov chain, intent-error table
Google or a one-time email code. No password.
