Problem
Alfa-Bank's call center first connects the client to a voice bot. A third-party ASR service transcribes the speech, and the team's classifier must pick the client's problem from 100+ topics. The old classifier was 10,000+ rules, "regexes on steroids". It ignores word semantics and gives no confidence score, so the automation versus satisfaction threshold cannot be tuned. A new rule for one topic can break another. With no labeled dataset, no metric could be computed after a change.
Example from the article: if the bot answers every question, automation rises, but more clients get a wrong answer and the satisfaction rating (Voice of Client, 1 to 5) falls. If it answers only when 100% sure, few calls close.
Solution in brief
Sample transcripts uniformly across the old rules' classes and label each with two annotators. Train a fastText plus bidirectional LSTM classifier. Compare it with the rules and TF-IDF baselines on a holdout, a historical slice and a small share of live customers.
Questions this case answers
- How to replace a rule-based topic classifier with thousands of rules by a neural model when there are no labels and no metric can be computed
- How to tune the trade-off between automation rate and satisfaction using a classifier that outputs class probabilities
- How to replace a rule-based call-routing classifier with a neural model
- How to build a labeled dataset when only unlabeled transcripts and an old rule-based classifier exist
Tools and methods
- fastText embeddings — input representation, robust to out-of-vocabulary words and recognition errors
- Bidirectional LSTM with max and average pooling, fully connected head — the classifier
- TF-IDF (word and character n-gram) — baselines
- Annotation with overlap 2 — keep examples where both annotators are confident
- Three-level model comparison — holdout, historical slice, online test on a share of customers
Google or a one-time email code. No password.