User-Guided LLM Answer Quality Evaluation

Ivan Bushmarinov23 May 2026

Problem

Perplexity is a small company that ships an answer engine to demanding users, and nobody can read the thousands of answers it produces daily. Answers are too complex to judge by clicks, and there are never enough people to validate each feature. Weak spots go unnoticed: a cheap model assigned to a "simple" scenario degrades answers, and offline tests miss what users really ask. The team needs a quality signal it can trust, and evals that product teams build without a data scientist.

Example from the article: Claude Haiku was the default model for a feature meant for small document edits. Users asked for full rewrites, and Haiku stood out as an outlier in dissatisfaction per model.

Solution in brief

An LLM judge scores every conversation for context loss, incompetence and user dissatisfaction from 0 to 10. The judge is validated against user dissatisfaction, split by plan price, not by hand reading. The same judges run offline through a terminal eval tool that coding agents drive.

Questions this case answers
  • How to check that an LLM judge can be trusted without reading answers by hand, using a signal that users produce for free
  • How to monitor LLM answer quality across a very large volume of production conversations
  • How to check that an LLM judge can be trusted without reading answers by hand
  • How to monitor LLM answer quality in production across thousands of daily conversations

Tools and methods

  • LLM judge — scores whole conversations daily; a low-thinking model is enough.
  • User dissatisfaction as ground truth — checked against the judge, split by plan.
  • pplx-eval CLI — runs a dataset through solvers and stacked judges.
  • Code, LLM and agent judges — deterministic checks, subjective grading, web search and code execution.
  • Prompt hill-climbing — an agent shortens a prompt against the eval.
Continue reading — it's free

Google or a one-time email code. No password.