Building Food Metadata with LLM Juries, Context Optimization & Multimodal AI

David Lindsay, Shuang Liu, Ying Yang2 Jul 2026

Problem

DoorDash needs structured attributes for millions of menu items and stores: is a dish spicy, which protein, what cuisine. Search, filters and personalization depend on them. Menus are written in every possible way and change daily. Few experts understand cuisines and menu wording, so human labeling and validation cannot cover the catalog. Handwritten prompts are brittle, and per-item API calls make a full backfill take over a month.

Example from the article: a tag such as "Sichuan-style" may or may not mean a heat profile, and telling Nepalese from North Indian dishes takes expert knowledge.

Solution in brief

An LLM generates tags from text, images and web search. A jury of several strong LLMs then votes on each tag separately, and the consensus replaces human validation. The jury's verdicts form an evaluation set. A tuning agent reads the failures on that set and rewrites the prompt rules. The same agents label data to fine-tune a small model, and Spark with batch APIs handles backfills.

Questions this case answers
  • How to evaluate model-generated labels at scale when expert human labeling is too scarce and too costly to cover the data
  • How to improve a prompt automatically against a fixed evaluation set instead of by manual trial and error
  • How to cut the inference cost of a large model by training a small model on labels produced without human annotators
  • How to check the quality of LLM-generated item tags when human experts cannot label millions of items?

Tools and methods

  • LLM jury — several LLMs judge each tag independently, votes are aggregated
  • Context-optimization agent — rewrites the prompt from failure cases
  • AI-led annotation — generation and evaluation agents produce labels for fine-tuning
  • Small fine-tuned model — replaces the large LLM at lower cost
  • Apache Spark and batch LLM APIs — deduplicated, parallel backfill
  • GEPA — population-based prompt optimizer, the rejected alternative
Continue reading — it's free

Google or a one-time email code. No password.