LLM in Education: How We Created the Code Guardian

Arseny Trushin25 May 2024

Problem

Yandex Uchebnik prepares school students for the Unified State Exam (EGE) in computer science, where many tasks are solved by writing Python or Kumir programs. Standard Python error messages do not help students see what they did wrong. The team wanted a "Help fix the error" button that returns a short, pedagogically correct hint without giving finished code. Prompts and few-shot examples were not enough, so it needed a new training dataset, a way to score free-form answers, and a path to production.

Example from the article: a student runs code that fails with a NameError, presses the button and gets a description of the error plus a recommendation for fixing it.

Solution in brief

Teachers wrote solutions, broken variants and ideal hints through Toloka, with a second review tier. About 10 thousand question-answer pairs were used to fine-tune an LLM. Quality was scored by hand on a fixed pool of 238 tasks, and a reward model trained on accepted and rejected annotations helped clean the dataset.

Questions this case answers
  • How to build a fine-tuning dataset from scratch with expert annotators and check it with an automatic verifier
  • How to find mislabeled records in a fine-tuning dataset with a reward model trained on accepted and rejected annotations
  • How to build an LLM feature that gives pedagogical hints for errors in student code
  • How to build a training dataset from scratch with teachers as annotators

Tools and methods

  • Fine-tuning — chosen after prompt engineering and few-shot examples proved insufficient
  • Synthetic dataset from teachers — about 100 teachers, three solutions per task
  • Toloka — annotation via an internal Yandex interface
  • Reward model — trained on good and bad answers to sort the dataset for cleaning
  • Manual pool evaluation — one point per appropriate, clear, correct answer
Continue reading — it's free

Google or a one-time email code. No password.