Problem
Yelp business pages hold reviews, photos, menus and structured facts, and users want a straight answer such as "Is the patio heated?" instead of reading dozens of reviews. A two-week prototype did this in demos but broke on real traffic. Users asked unsafe and out-of-scope questions. Answers were wordy, sometimes contradicted the evidence, took 10–20 seconds and cost too much per question. Productizing it took nine months.
Example from the article: asked for hikes for 3–5 year old kids, the prototype recommended the 5+ mile Mirror Lake Trail. The production version dropped it and gave distances and durations.
Solution in brief
Small fine-tuned models classify and route each question before any evidence is fetched, and the answer is then generated with citations and streamed. Three LLM judges score answers on correctness, completeness and evidence relevance, so prompts and cheaper models can be changed safely.
Questions this case answers
- How to move a grounded question-answering assistant from a demo to production when real users ask unsafe, out-of-scope and unexpected questions
- How to reduce latency and token cost of a retrieval-augmented assistant while a quality grader guards against regressions
- How to evaluate open-ended generated answers on several quality criteria when only a few hundred human labels exist
- How to turn a two-week LLM question-answering prototype into a production system that answers from a business page's own content?
Tools and methods
- Question analysis models — small fine-tuned models for safety, inquiry type, source selection and keywords, run in parallel
- Keyword retrieval — near-real-time indices for reviews, photos and websites; Cassandra for structured facts
- LLM-as-a-judge graders — one per criterion, tuned on hand-labeled answers
- Langfuse — stores daily grading results, cost and tokens
- Streaming (FastAPI, server-sent events) — sends the answer as it is generated
- Dynamic prompt composition — only the instructions and few-shot examples the question needs
Google or a one-time email code. No password.