Problem
Avito support handles 3 million contacts a month with 2,500 agents, and most of the work is repetitive and manual. The team wants shorter handling time (AHT) and higher customer satisfaction (CSAT). LLM features for agents are hard to ship. Generative datasets depend on many business wishes, quality has no obvious metric, and agents must find the right guideline among thousands.
Example from the article: a button turns a rude draft ("You've worn me out! I've already told you a hundred times…") into a polite answer with the same meaning. The first guideline retriever had the right one in its top-5 only 34% of the time.
Solution in brief
Build datasets against written product requirements and score outputs by expert labeling on explicit criteria. For guideline search, blend BM25 with two dense retrievers, then match extracted "problems" of the chat and the guideline instead of whole documents.
Questions this case answers
- How to evaluate generative text features when no ground-truth label exists and quality depends on written requirements
- How to improve retrieval of the right instruction for a free-text conversation by matching extracted problems rather than whole documents
- How to make LLM features for support agents (rephrasing, chat summaries, guideline suggestions) pay off in AHT and CSAT
- How to evaluate generative text-to-text features when there is no ground-truth label
Tools and methods
- Fine-tuned 7B LLM with LoRA — message rephrasing and chat summaries
- Expert labeling on 3 and 5 criteria — offline quality of generative outputs
- A/B test on AHT — online effect of chat summaries
- multilingual-e5-base, rubert-tiny2, e5-mistral-7b-instruct, BM25 — retrievers for guidelines
- Ensemble, chunk-position reorder, roberta reranker — ranking of retrieved chunks
- LLM problem extraction (DeepSeek, LoRA adapters) — matching chat problems to guideline problems
Google or a one-time email code. No password.