Problem
Avito sells cars through products where a manager calls the client and then hands them to a dealer. A good call follows a checklist (introduce yourself, greet, agree on a meeting), and better calls raise the chance of a deal. Manual labeling covers only 5% of calls and is slow and expensive. The contractor's model was cheaper, but its quality was not always good. Checklists differ between products and dealers and keep changing, so a model must handle criteria it has not seen.
Example from the article: a transcript plus the criterion "the manager introduced themselves and greeted the client" goes in, and "yes" or "no" comes out.
Solution in brief
Fine-tune an LLM that takes a call transcript and one checklist criterion and answers yes or no, a single token each. The team compared per-checklist LoRA adapters with one general model trained on all checklists, and served the model with vLLM. The same recipe (large-LLM labeling with a validated prompt, LoRA training, vLLM service) was reused for support-reply rewriting and Autoteka abbreviation decoding.
Questions this case answers
- How to automate yes-or-no checks of long conversations against a set of changing criteria with a small fine-tuned LLM
- How to train a small model with a LoRA adapter on labels produced by a large LLM with a long validated prompt when no labels exist
- How to automate call-quality checklist evaluation with a fine-tuned LLM
- How to train a small LLM with LoRA on labels generated by a large LLM with a long prompt
Tools and methods
- LoRA adapters — fine-tuning on a base LLM, per task or per checklist.
- vLLM — model serving.
- LLM-based labeling — a large LLM with a long prompt generates labels.
- Redis cache — keyed with a model-version check.
- Regex post-processing — output rules enforced after generation.
Google or a one-time email code. No password.