Problem
Sber Devices builds a dialogue system that answers by retrieval: it embeds the last utterances and picks the best answer from a bank of candidates. This model barely kept up with generative GPT-like models because its answers lacked specificity. Human feedback that could fix this is expensive: experts must write or annotate dialogues for every topic.
Example from the article: a context of six utterances with one good answer, one random bad answer and one hand-written hard bad answer. The good one must win.
Solution in brief
Train a reward model (a critic) on expert-labeled good and bad dialogues, then use its 0 to 1 score in three ways: re-rank candidates, widen the margin of the triplet loss for good pairs, and label self-generated dialogues. The same critic can also drive actor-critic RL (DDPG, then PPO) over embeddings.
Questions this case answers
- How to improve a retrieval model that picks the best answer from a candidate pool using a learned critic score instead of more human labels
- How to train an embedding retrieval model with a critic-adjusted triplet margin
- How to improve a retrieval-based dialogue system using a reward model instead of more human labels
- How to use a critic score inside triplet loss to train better context and answer embeddings
Tools and methods
- ruRoBERTa encoder — multitask base encoder, fine-tuned with metric learning
- State Context Understanding (SCU) — small model: six utterance vectors in, one context vector out
- Reward model (ruElectra) — critic scoring a context plus answer from 0 to 1
- Triplet loss with adaptive soft margin — the critic score is added to the margin
- Black-box adversarial augmentation — synonym swaps that hurt the model most
- DDPG and PPO — actor-critic training for retrieval
- Multi-armed bandit — picks which retrieval sources to call per query
Google or a one-time email code. No password.
