Socratic Dialogues with LLMs in Problem Solving

Dmitriy Yudochev25 May 2024

Problem

An education platform wants an LLM assistant that teaches students to solve problems instead of acting as a "problem solver" that teachers fear. A plain LLM tends to give the full solution, drifts from the task, confuses steps and accepts wrong answers. The audience is children, so unsafe replies carry legal and reputational risk. Yandex Uchebnik faced this in its free Unified State Exam (EGE) Informatics preparation platform.

Example from the article: a student clicks "Suggest a solution". The control group sees a text outline of the solution. The test group gets a question ("how many seconds does the audio file last?") and a guided dialogue.

Solution in brief

Methodologists wrote a step-by-step script (question, line of reasoning, correct answer) for every task subtype. Each reply comes from a four-part prompt: role, task, script, dialogue so far. YandexGPT was prompt-tuned on about 140 dialogues, with ethics classifiers as filters. In an A/B test, answer submission rose from 46% to 61% and correct solutions from 69% to 88%.

Questions this case answers
  • How to make an LLM assistant guide a learner to the answer through questions instead of giving the solution, while following an expert-written script and staying safe for children
  • How to choose between prompt-only, few-shot, fine-tuning and prompt-tuning adaptation for a dialogue assistant when examples are limited
  • How to make an LLM tutor guide a student to the answer instead of handing over the solution
  • How to make an LLM follow a step-by-step script without prompt engineering by trial and error

Tools and methods

  • Solution script — per task subtype: question, reasoning for hints, correct answer
  • Four-part prompt — role, task, script, dialogue so far
  • Prompt-tuning (P-tuning) — a small trained add-on; the base model stays frozen
  • Ethics classifiers — filters on user and assistant phrases
  • A/B test — 50/50 split of users who clicked "Suggest a solution"
Continue reading — it's free

Google or a one-time email code. No password.