Learning to Understand Dialogs Based on Real Data

Anton Legchenko28 Oct 2023

Problem

A money-transfer service (CFT, the "Zolotaya Korona" brand) runs a support chatbot with more than 5000 completed dialogs a day, in seven languages including Uzbek, Tajik and Kyrgyz. The intent classifier is a two-year-old BERT model that sees only the last message. It was trained on 100k+ single messages with noisy labels. With 150+ intents that change often, every new class means retraining the whole model.

Example from the article: a customer asks for an operator, the operator replies "let me review the dialog," and the real question sits earlier, so the last message carries no intent.

Solution in brief

Train an encoder on 1M+ unlabeled historical dialogs with two self-supervised tasks: "are these two parts from the same dialog" and "similar dialogs get similar operator answers." Then train a light classifier on a small clean labeled set. In context the combined encoder reaches F1-macro 69 (frozen) against 47 for the old BERT.

Questions this case answers
  • How to train an intent classifier for multi-turn conversations that uses the whole dialog context when labeled data is small and the set of classes changes often
  • How to pretrain a multilingual dialog encoder on a large set of unlabeled historical conversations with self-supervised tasks
  • How to train a multilingual dialog encoder for intent classification with little labeled data
  • How to make a support-chat intent classifier use dialog context instead of the last message only

Tools and methods

  • mT5-base encoder — covers all required languages (102); XLM-RoBERTa misses two of them.
  • Mixture of denoisers (UL2 / FRED-T5 style) — domain adaptation on dialog text, with masking spans confined to one message.
  • Dialog-pair task — the encoder decides if two dialog parts come from one dialog.
  • Operator-answer similarity — match cosine-distance matrices of dialog contexts and of operator answers.
  • Frozen encoder plus light classifier — the evaluation setup, and the production recommendation.
Continue reading — it's free

Google or a one-time email code. No password.