Problem
Yandex Crowd support answers tickets about promo codes: a user took part in a promotion, did not get the code and believes all conditions were met. An operator checks admin panels, earlier tickets and the knowledge base, then picks an answer template. The flow is a rigid tree of "if A then B" rules, and a plain RAG bot reached only 40% quality.
Example from the article: a user used Yandex search every day for 14 days under a "600 rubles for a week of search" promotion, yet the promotion page says the conditions are not met.
Solution in brief
Replace RAG with a 12-node LangGraph workflow: pre-checks, admin-panel data, a ReAct agent that walks the knowledge base as a tree, and a judge that answers or hands over to an operator. Constrain reasoning with Schema-Guided Reasoning (SGR) and debug through Langfuse traces. Quality rose from 40% to 74% on labeled tickets.
Questions this case answers
- How to make an LLM agent pick the right answer template when the decision flow is a rigid tree and plain retrieval reaches low answer quality
- How to make LLM reasoning structured, controllable and traceable in production
- How to make an LLM support agent reliable when plain RAG only reaches about 40% answer quality
- How to pick the right knowledge-base article and answer template when the support flow is a rigid decision tree
Tools and methods
- LangGraph — graph of nodes, edges and shared state, with persistence.
- ReAct agent — navigates the knowledge base through list, table-of-contents and read-section tools.
- Schema-Guided Reasoning (SGR) — each reasoning step goes into a field of a JSON schema (structured output).
- LLM judge tuned with DSPy — checks the final answer; failing tickets go to an operator.
- Langfuse — traces, prompt management, dashboards.
- Lightweight OCR — reads ticket screenshots.
Google or a one-time email code. No password.
