Improving the reliability of generative models: an approach to evaluating responses and data preparation

Aleksey Yandutov28 Oct 2023

Problem

The Facts team at Yandex builds generative fast answers for search with YandexGPT. A model answering "from its head" hallucinates plausibly, especially on topics where the user is not an expert. Even when it must answer only from retrieved documents, it still lies on complex queries, long unclear documents, and when the document has no answer. Without a measure of "supported by the source" the team could neither compare model versions nor train for reliability.

Example from the article: for "5 reasons to choose handmade soap" the document did not list five reasons, so the model invented a last one about soap being cheaper because it needs no synthetic components.

Solution in brief

Split each generated answer into statements and have assessors check every statement against the source with a seven-category scheme. Fix the consistency problems of the labeling (overlap of three, staged instructions, training with an exam, automatic honeypots) and reuse the labels to train a reliability classifier that also serves as a reward.

Questions this case answers
  • How to measure whether a generated text is supported by its source documents statement by statement when human fact-checking is the only reliable judge
  • How to design a multi-category human labeling scheme for checking generated text against a source and keep crowd assessors consistent
  • How to measure whether LLM answers in a search product are supported by their source documents
  • How to design a human labeling scheme for fact-checking generated answers

Tools and methods

  • Closed fact-checking — answer checked against the documents the model saw, about 7 minutes
  • Seven-category statement labeling — confirmed, does not require verification, sources disagree, could not confirm, distorts meaning, absurd, contradicts
  • Overlap of 3 with aggregation — three assessors per task, merged into two classes
  • Dynamic honeypots — automatically collected control tasks, automatic bans
  • Token-level, segment-level and answer-level classifiers — trained on the labels
Continue reading — it's free

Google or a one-time email code. No password.