AgentOps in Production: Infrastructure for LLM Agents and Support Automation for Yandex Personal Services

Ivan Nasonov23 May 2026

Problem

Support for Yandex content services (Kinopoisk, Music, Books, Yandex Plus and others) gets about 75 thousand requests a day, and operators carry a heavy load of subscription and refund cases. After moving from button graphs to ML classifiers and LLM agents, two things limited progress. LLM serving had to answer within 5 seconds at P90, survive the loss of a data center and keep internal experiments from starving production. And every new agent needed ML, backend and tester time.

Example from the article: after a subscription price rise, load reached 10 RPS, high for this service, and answers had to keep coming.

Solution in brief

Requests pass through a funnel: ML classifier, LLM classifier for hard cases, a ready graph or an LLM agent with RAG, then an operator. Inference runs in two data centers behind LiteLLM (billing, keys, priorities) and HAProxy (routing, sticky sessions for KV-cache). Product managers build agents themselves in the internal no-code tool ConclaveAI.

Questions this case answers
  • How to build serving infrastructure for LLM agents that must answer within a strict P90 latency, survive loss of a data center and keep internal experiments from starving production
  • How to route incoming service requests by difficulty between cheap classifiers, LLM classifiers, LLM agents and human operators
  • How to build the serving infrastructure for LLM support agents so that P90 latency stays within 5 seconds
  • How to route customer support requests between ML classifiers, LLM classifiers, LLM agents and human operators

Tools and methods

  • ML classifier (CatBoost, embedders 0.5–1.5B) — fast topic prediction from the first message
  • LLM classifier — fallback that sees the whole dialogue context
  • Tone of Voice agent — light LLM that checks and rewrites the final answer
  • LiteLLM Gateway — keys, token billing, priority rate limits
  • HAProxy — L7 balancing by X-Model-Name, leastconn or sticky sessions
  • ConclaveAI — no-code agent and workflow builder
Continue reading — it's free

Google or a one-time email code. No password.