LLM-as-a-Judge: Evaluating natural language search

Xiaochang Miao, Heather Song14 May 2026

Problem

DoorDash's natural language search handles queries such as "cozy date night dinner" or "gluten-free fast spicy affordable". These queries have no click history, and click-through and null-rate metrics are lagging and noisy. Human relevance labeling took two to five days per cycle, and the labels were inconsistent, so every change to query rewriting, retrieval or ranking waited on a weak quality signal.

Example from the article: a reranker fine-tuned on the human labels reached an AUC of 0.56 on a held-out set, barely above random. The labels were the problem, not the model.

Solution in brief

Split relevance into independent yes/no checks over intent facets (dish, price, speed and so on), answerable from observable store and menu data. Calibrate an LLM judge on an adjudicated golden set, then run it daily on production traces and on every search-pipeline pull request, with metrics reported per facet.

Questions this case answers
  • How to evaluate the relevance of ranked results for free-form, multi-constraint queries when click logs do not cover them and human labels are slow and inconsistent
  • How to calibrate an automated LLM judge against noisy human annotations and keep it calibrated as the product changes
  • How to evaluate the relevance of natural-language search results when there is no click ground truth for the query?
  • How to build a rubric that an LLM judge and human raters apply the same way?

Tools and methods

  • LLM-as-a-judge — grades each query-store pair facet by facet and returns a rationale
  • Binary facet rubric — replaces a 0 / 0.5 / 0.8 / 1.0 scale with yes/no checks
  • Golden set — adjudicated labels, the calibration target for the judge
  • Per-facet NDCG@5 — localizes failures to one facet
  • Pull-request guardrail — sandbox replay, control against treatment, pass/fail
  • Qwen3-based reranker — the supervised model that exposed the label noise
Continue reading — it's free

Google or a one-time email code. No password.