How DoorDash leverages LLMs to evaluate search result pages

Yulei Liu30 Apr 2025

Problem

A search team needs relevance judgments for millions of query-result pairs, and human annotators cannot supply them. Feedback takes days or weeks, raters read the guidelines differently, and labeled sets cover mostly frequent (head) queries while relevance problems hide in the tail. A results page also mixes stores, dishes and item carousels, so a single-list metric misses what the user sees. DoorDash hit this as search grew from restaurants to retail, grocery and pharmacy.

Example from the article: offline experiments on new ranking models, filters and UI changes waited on human labeling, which became the bottleneck.

Solution in brief

DoorDash built AutoEval: sampled production queries are turned into structured prompts, an LLM (base or fine-tuned) returns a relevance judgment for each result, and a page-level whole-page relevance (WPR) score weights judgments by block position. Experts, external raters and fine-tuning keep the judge aligned with humans.

Questions this case answers
  • How to automate relevance evaluation of a multi-block results page at scale with LLM judges that stay aligned with human raters
  • How to evaluate a ranking change offline, before an online A/B test, when human labeling is too slow and covers mostly head queries
  • How to replace slow human relevance labeling with LLM judges without losing agreement with human raters?
  • How to score a whole search page with several blocks (stores, dishes, items) instead of one ranked list?

Tools and methods

  • LLM-as-a-judge (GPT-4o via OpenAI API) — rates each query-result pair against the human rubric
  • Whole-page relevance (WPR) — page-level score that weights content blocks by layout position
  • Chain-of-thought prompts with embedded guidelines — multi-step rating logic and domain context in the prompt
  • Supervised fine-tuning on a golden dataset — expert labels with justifications train and benchmark the judge
  • Human-in-the-loop audit — external raters flag bad judgments, experts fix prompts and labels
Continue reading — it's free

Google or a one-time email code. No password.