Problem
A call center of one CFT service has chat and telephony. Operators label topics in chats but rarely in calls, so nobody knows what customers call about. A later task was to check whether sales and support operators follow a script of 10-12 points, such as showing initiative or handling objections. Dialogues are long, the points are abstract, and only a few hundred labeled dialogues exist. A BERT model trained on chats and applied to ASR text of calls reached F1 of only about 70%.
Example from the article: a customer hangs up early, so the operator never had a chance to handle objections. Business rules say such a point is "yes", while an untuned model says "no".
Solution in brief
For topics, an LLM summarizes every chat and call into one short line about what the customer wanted, and BERT classifies those summaries. For script checks, a 7B LLM is fine-tuned with QLoRA to answer in text, one answer per script point. Business logic that the model misses is added as if rules on top of auxiliary zero-shot questions.
Questions this case answers
- How to classify the topic of long multi-turn conversations when labels exist only for short ones, by summarizing each conversation into one short line first
- How to check a long dialogue against a multi-point checklist with a few hundred labeled examples, by fine-tuning an open LLM on a single small GPU
- How to classify the topic of long phone calls when labels exist only for chats
- How to check call-center operator scripts on long dialogues with only 200-1000 labeled examples
Tools and methods
- Whisper Large V3 — speech recognition for calls
- LLM summarization (Llama) — one short format for chats and calls
- BERT — topic classifier on summaries
- QLoRA / LoRA — fine-tune a 4-bit quantized LLM on one GPU
- Mistral 7B OpenChat — base model for script checking
Google or a one-time email code. No password.