NLP at Alfa-Bank, Part 4: The Development History of a Chatbot

Maxim27 May 2023

Problem

A bank's support chatbot answers from a fixed scenario tree, so each reply depends on the right intent and entities. Alfa-Bank's team started with a labeled set of 150 thousand messages, and LogReg, LSTM and BERT all scored about the same on it. The team suspected the set was easier than the real stream. A wrong intent either sends the session to a call-center operator (automation drops) or gives the client a wrong answer (the average rating drops).

Example from the article: a client wrote "fees for withdrawing from a card"; the bot found no card type, asked a clarifying question and then reached the final answer.

Solution in brief

Compare the training set with the whole message stream (bigram coverage plus a classifier that separates the two), then build a labeling process. Pick the model by production limits and tune the automation/rating balance with thresholds. Close the loop: per-intent monitoring sends weak spots to assessors, who fix the data before retraining.

Questions this case answers
  • How to check whether an intent classifier's training set reflects the real stream of incoming messages before tuning models
  • How to balance automation rate against user rating by setting classifier confidence thresholds, and keep the training data improving through per-class monitoring
  • How to find out that an intent-classification training set does not reflect the real message stream
  • How to set up data labeling for a customer-support chatbot so that model quality keeps growing

Tools and methods

  • Bigram coverage and train-vs-stream classifier — check that the training set represents the stream.
  • LSTM with FastText, BERT — intent models, both pre-trained on 200 million bot messages.
  • Cross-labeling, uncertainty sampling — label quality and choice of messages to label.
  • BERT like model — predicts whether the client will like a reply.
  • BERT trained on regex labels — entity extraction without manual labeling.
Continue reading — it's free

Google or a one-time email code. No password.