Tackling AI Hallucinations in LLM Apps

Denis Kazakov17 Feb 2025

Problem

A support team wants an LLM to answer customer questions from internal documentation, for example "How do I add a contractor?". Sometimes the model answers well. Other times it makes things up or gives a vague reply. Classic ML models return a class and a confidence, so a threshold can trade coverage against precision. LLM output is free text, so that control seems unavailable and bad answers reach customers.

Example from the article: Gusto ran 1000 support questions through its question-answering LLM service, and support experts labeled each answer as good or bad.

Solution in brief

Use the average log-probability of the generated tokens (Seq-Logprob) as the answer's confidence. It comes free with generation. Label a sample of answers, check that confidence tracks quality, and set a threshold. Answers below it are rejected, sent to an expert, or retried with more information.

Questions this case answers
  • How to cheaply estimate whether a free-form generated answer is reliable, when there is no class label or native confidence score to threshold
  • How to trade answer coverage against answer quality for a generative system using a threshold on a by-product signal
  • How to stop low-quality LLM answers from reaching customers without a second model call?
  • How to get a precision-recall trade-off for a free-text LLM feature?

Tools and methods

  • Seq-Logprob (sequence log-probability) — mean log-probability of the generated tokens, used as the confidence score
  • Precision-recall curve — shows the coverage versus quality trade-off for each confidence threshold
  • Expert labeling — support experts mark answers good or bad to calibrate the threshold
  • OpenAI API logprobs — source of the token log-probabilities
Continue reading — it's free

Google or a one-time email code. No password.