Development History of Voice Bot: Legacy Voice Bot of Alfa-Bank and Transition to Modern Methods

Vladislav Galichanskiy27 May 2023

Problem

Alfa-Bank's call center first connects the client to a voice bot. A third-party ASR service transcribes the speech, and the team's classifier must pick the client's problem from 100+ topics. The old classifier was 10,000+ rules, "regexes on steroids". It ignores word semantics and gives no confidence score, so the automation versus satisfaction threshold cannot be tuned. A new rule for one topic can break another. With no labeled dataset, no metric could be computed after a change.

Example from the article: if the bot answers every question, automation rises, but more clients get a wrong answer and the satisfaction rating (Voice of Client, 1 to 5) falls. If it answers only when 100% sure, few calls close.

Solution in brief

Sample transcripts uniformly across the old rules' classes and label each with two annotators. Train a fastText plus bidirectional LSTM classifier. Compare it with the rules and TF-IDF baselines on a holdout, a historical slice and a small share of live customers.

Questions this case answers
  • How to replace a rule-based topic classifier with thousands of rules by a neural model when there are no labels and no metric can be computed
  • How to tune the trade-off between automation rate and satisfaction using a classifier that outputs class probabilities
  • How to replace a rule-based call-routing classifier with a neural model
  • How to build a labeled dataset when only unlabeled transcripts and an old rule-based classifier exist

Tools and methods

  • fastText embeddings — input representation, robust to out-of-vocabulary words and recognition errors
  • Bidirectional LSTM with max and average pooling, fully connected head — the classifier
  • TF-IDF (word and character n-gram) — baselines
  • Annotation with overlap 2 — keep examples where both annotators are confident
  • Three-level model comparison — holdout, historical slice, online test on a share of customers
Continue reading — it's free

Google or a one-time email code. No password.