Modern training methods for retrieval-based dialogue models

Alexander Abramov4 Jun 2022

Problem

A voice assistant (Salute at SberDevices) answers free conversation with a retrieval model. It picks the best reply among editor-written ones, using the last six phrases of the dialogue as context. One core encoder serves about 10 tasks at once (paraphrasing, dialogue modeling, NER, intents, sentiment). With 2 million training samples and a 32 GB V100, only 24–32 samples per task fit into a batch, and BERT-like models depend strongly on batch size. The embeddings are also fragile to typos and synonyms, and the index is large.

Example from the article: the dialogue model embeds a six-phrase context and ranks editor-written replies by recall@top-k.

Solution in brief

Train the encoder with triplet loss on context and reply vectors, and add four tricks: weight averaging across GPUs, distillation of LaBSE embeddings, adversarial text augmentations as extra positives, and "full batching" with all other triples as negatives. A critic score from labeled data is added to the triplet margin.

Questions this case answers
  • How to train a text embedding model for ranking candidate replies when many simultaneous tasks leave only tiny per-task batches
  • How to shrink a text embedding index while keeping ranking relevance, using a larger multilingual model as the teacher
  • How to make text embeddings stable under typos and synonym substitutions in user input
  • How to train a retrieval-based reply ranker for a voice assistant when only 24–32 samples per task fit into a batch?

Tools and methods

  • Triplet loss — pulls the context toward the relevant reply and away from irrelevant ones.
  • Weight averaging — twin networks trained on different shards, weights averaged.
  • Knowledge distillation — MSE between compressed student vectors and LaBSE vectors.
  • Data augmentation — a "broken" query becomes a positive for the original.
  • In-batch negatives — positives of other triples serve as negatives.
  • LaBSE, RoBERTa, Horovod — teacher, student, multi-GPU training.
Continue reading — it's free

Google or a one-time email code. No password.