Problem
Support for Yandex content services (Kinopoisk, Music, Books, Yandex Plus and others) gets about 75 thousand requests a day, and operators carry a heavy load of subscription and refund cases. After moving from button graphs to ML classifiers and LLM agents, two things limited progress. LLM serving had to answer within 5 seconds at P90, survive the loss of a data center and keep internal experiments from starving production. And every new agent needed ML, backend and tester time.
Example from the article: after a subscription price rise, load reached 10 RPS, high for this service, and answers had to keep coming.
Solution in brief
Requests pass through a funnel: ML classifier, LLM classifier for hard cases, a ready graph or an LLM agent with RAG, then an operator. Inference runs in two data centers behind LiteLLM (billing, keys, priorities) and HAProxy (routing, sticky sessions for KV-cache). Product managers build agents themselves in the internal no-code tool ConclaveAI.
Questions this case answers
- How to build serving infrastructure for LLM agents that must answer within a strict P90 latency, survive loss of a data center and keep internal experiments from starving production
- How to route incoming service requests by difficulty between cheap classifiers, LLM classifiers, LLM agents and human operators
- How to build the serving infrastructure for LLM support agents so that P90 latency stays within 5 seconds
- How to route customer support requests between ML classifiers, LLM classifiers, LLM agents and human operators
Tools and methods
- ML classifier (CatBoost, embedders 0.5–1.5B) — fast topic prediction from the first message
- LLM classifier — fallback that sees the whole dialogue context
- Tone of Voice agent — light LLM that checks and rewrites the final answer
- LiteLLM Gateway — keys, token billing, priority rate limits
- HAProxy — L7 balancing by
X-Model-Name,leastconnor sticky sessions - ConclaveAI — no-code agent and workflow builder
Google or a one-time email code. No password.
