An AI Agent as Teacher and Examiner

Alexey Golovin23 May 2026

Problem

Companies train staff with a chain of materials, courses and tests. That gives knowledge but not practice, and real conversations stay outside the training. Methodologists write practice scenarios by hand, so updating them is slow. HR sees that a course was completed but not how ready a person is for a real dialogue. Errors show up only after complaints.

Example from the article: Cerevrum (clients in retail, FMCG, banks, telecom) turns a methodology document into a practice dialogue such as "selling realtor services", where an LLM plays the apartment owner and the employee must win his agreement to a meeting.

Solution in brief

Parse source documents into a retrieval layer (vectors plus an entity graph). From it, generate a lesson, knowledge-check questions and a practice scenario with a goal, an agent script and scoring criteria. A methodologist edits the draft. The LLM then plays the role in a dialogue and scores the user by the criteria. RAG and the agent are tested separately offline.

Questions this case answers
  • How to generate practice scenarios and knowledge checks from an organization's own documents with a human expert in the loop
  • How to answer questions whose evidence is spread over several document sections by linking entities and topics to chunks in a graph
  • How to generate training scenarios and practice dialogues from company documents with an LLM
  • How to build an LLM role-play agent that stays on a scenario and scores the user

Tools and methods

  • RAG and GraphRAG — find the context for explanations, questions and answers.
  • Practice scenario editor — exposition, goal, agent script, criteria, role.
  • LLM and keyword indicators — score competencies from the dialogue.
  • RAGAS metrics — Faithfulness, Answer Relevance, Context Relevance.
  • Checklist autotests — check that the agent follows the scenario.
Continue reading — it's free

Google or a one-time email code. No password.