An AI Tutor and Methods for Evaluating It

Olga Masaeva, Polina Poveteva23 May 2026

Problem

Sea of Data, a small edtech startup, runs an LLM tutor for schoolchildren and corporate courses. The tutor should lead a student to the answer with guiding questions (Socratic dialogue), not hand it over. Generic LLM benchmarks do not show whether it does. The team must also know how it reacts to wrong answers and whether students can talk it into revealing the answer. There is no labeled data, and expert annotators have little time.

Example from the article: on 7th-grade physics sessions, the tutor's Telling Rate was 0.65, against 0.16 for the "Teacher of the Year" finalists.

Solution in brief

The authors evaluate on three levels. A classifier trained on labeled teacher lessons tags each tutor move, and the tags aggregate into Socratic metrics compared with expert teachers. Single replies are scored by the type of the student's answer. LLM-simulated students try to extract answers.

Questions this case answers
  • How to evaluate whether a dialogue assistant guides the learner through reasoning instead of giving the answer, turn by turn
  • How to build a labeled dataset of dialogue moves cheaply when expert annotators have little time
  • How to evaluate the pedagogical quality of an LLM tutor's Socratic dialogue turn by turn
  • How to measure whether a tutor guides students or just tells them the answer

Tools and methods

  • TalkMoves-style move classifier — fine-tuned on 75,768 labeled sentences; 11 move classes.
  • Socratic dialogue metrics — Telling Rate, Socratic Ratio, PMR, MCL, IRF/IRE ratio, aggregated from the tagged moves.
  • Behavioral evals by student answer type — five criteria per tutor reply.
  • LLM-as-a-judge pre-labeling — confidence from logits orders the annotators' queue.
  • LLM student personas — simulated schoolchildren with different characters that try to extract answers.
Continue reading — it's free

Google or a one-time email code. No password.