Using LLM to transcribe restaurant menu photos

Zhe Mai, Zheng Hu, Ying Yang19 Mar 2025

Problem

DoorDash keeps restaurant menus up to date from photos that merchants submit. Humans transcribed them, which was costly and slow. An OCR step followed by an LLM gives a quick structured draft, but the required accuracy is very high and the LLM misses it on a noticeable share of menus: wrong item names, wrong categories, wrong item-to-price links. Pushing the LLM itself to the target accuracy would take a long time and delay any value. The cost is either wrong menus in production or a fully manual process.

Example from the article: a photo where item names and descriptions span a varying number of lines, so the LLM attaches a price to the wrong item.

Solution in brief

Treat the LLM as one component and put a small ML guardrail model after it. The guardrail predicts, per photo, whether the transcription is accurate enough. Accepted transcriptions go live and the rest go to the human process. The guardrail stays the same when the transcription model is replaced.

Questions this case answers
  • How to automate an extraction task with an LLM when the required accuracy is higher than the model reaches on a hard tail of inputs
  • How to decide per item whether a model output can be trusted or must be routed to a human, using a separate classifier
  • How to put an LLM data-extraction pipeline into production when errors are costly and the model cannot be made accurate enough alone?
  • How to predict whether a single LLM transcription is accurate enough to skip human review?

Tools and methods

  • OCR + LLM transcription — extracts text from a menu photo and turns it into structured items.
  • Guardrail model — binary classifier that predicts whether a transcription meets the accuracy requirement.
  • LightGBM — the best guardrail model and the fastest to run.
  • ResNet, ViT (and VGG16, DiT) — pre-trained image encoders tried inside a neural-network guardrail.
  • Human fallback — manual transcription for photos the guardrail rejects.
Continue reading — it's free

Google or a one-time email code. No password.