RLHF for retrieval based NLP system

Alexander Abramov27 May 2023

Problem

Sber Devices builds a dialogue system that answers by retrieval: it embeds the last utterances and picks the best answer from a bank of candidates. This model barely kept up with generative GPT-like models because its answers lacked specificity. Human feedback that could fix this is expensive: experts must write or annotate dialogues for every topic.

Example from the article: a context of six utterances with one good answer, one random bad answer and one hand-written hard bad answer. The good one must win.

Solution in brief

Train a reward model (a critic) on expert-labeled good and bad dialogues, then use its 0 to 1 score in three ways: re-rank candidates, widen the margin of the triplet loss for good pairs, and label self-generated dialogues. The same critic can also drive actor-critic RL (DDPG, then PPO) over embeddings.

Questions this case answers
  • How to improve a retrieval model that picks the best answer from a candidate pool using a learned critic score instead of more human labels
  • How to train an embedding retrieval model with a critic-adjusted triplet margin
  • How to improve a retrieval-based dialogue system using a reward model instead of more human labels
  • How to use a critic score inside triplet loss to train better context and answer embeddings

Tools and methods

  • ruRoBERTa encoder — multitask base encoder, fine-tuned with metric learning
  • State Context Understanding (SCU) — small model: six utterance vectors in, one context vector out
  • Reward model (ruElectra) — critic scoring a context plus answer from 0 to 1
  • Triplet loss with adaptive soft margin — the critic score is added to the margin
  • Black-box adversarial augmentation — synonym swaps that hurt the model most
  • DDPG and PPO — actor-critic training for retrieval
  • Multi-armed bandit — picks which retrieval sources to call per query
Continue reading — it's free

Google or a one-time email code. No password.