Exploring How to Improve Assessment with AI

Kristen DiCerbo27 Oct 2025

Problem

Many see large-scale tests as limited to question types that can be scored automatically, so they measure less than they should. A teacher would ask "Tell me why you did that step", but a single written answer rarely shows that. An LLM in assessment adds two risks: scores may disagree with human raters, and a helpful model may give away hints. Writing items and scoring criteria and testing them on real students can also take months to years per assessment.

Example from the article: Khan Academy piloted Explain Your Thinking, where a student answers an algebra or geometry question and then talks to an AI about the answer.

Solution in brief

The student answers a standard question, then the AI asks follow-ups aimed at specific concepts. An AI scorer checks the criteria after each turn and gives a final score. Self-critique keeps the AI from hinting, and synthetic responses let the team tune items and criteria before any student pilot.

Questions this case answers
  • How to validate an LLM judge of multi-turn conversations against human raters when labeled conversations are scarce
  • How to prevent a helpful-by-default LLM from revealing information it must withhold in a test-like setting
  • How to tune scoring criteria for open-ended items before expensive human pilots by simulating responses
  • How to check that an LLM scorer of student conversations agrees with human raters?

Tools and methods

  • Conversational follow-up — a prompted AI asks about the student's reasoning after the first answer
  • AI scorer — judges per-criterion correct/incorrect after every turn, then scores the whole conversation
  • Self-critique before display — the AI reviews its own reply before the test-taker sees it
  • Synthetic response generation — 150 simulated student responses per item for pre-pilot iteration
Continue reading — it's free

Google or a one-time email code. No password.