Problem
A team ships an LLM agent that turns business questions into SQL and charts. When the agent misreads the metadata, the database still runs the query and the dashboard looks fine, so a wrong number reaches a decision. There is no syntax error to catch it. Correctness is not binary, answers vary between runs, and any model or speed change can break a report that was right.
Example from the article: the Genie team at Databricks must not let a speed optimization break a critical revenue report. A CEO asking for revenue growth for May 2026 must get the same answer every time, because a roughly similar number counts as wrong.
Solution in brief
Build the eval set from production traffic, with ground truth from public benchmarks, internal experts and external vendors. Use an LLM judge to compare versions, not to give absolute scores. Run benchmarks daily on the production code and alert when the score drops below a threshold. Feed production failures back into the set.
Questions this case answers
- How to build an evaluation set for a text-to-SQL agent from production traffic and keep it current when correctness is not binary and answers vary between runs
- How to use an LLM judge to compare versions of an agent when the judge is more lenient than human graders and cannot give reliable absolute scores
- How to build an evaluation set for an LLM agent that answers business questions with SQL
- How to use an LLM judge when it grades more generously than humans
Tools and methods
- Production traffic study — classifies queries by workflow type and difficulty level.
- Three-source ground truth — Spider and BIRD, internal experts, external blind audits.
- LLM-as-judge — grades complex answers; used for pairwise comparison.
- Golden-query regression guard — over 2,000 queries run daily, with alerts.
- Production signals — customer-reported failures, exports and shares, thumbs up and down.
- Confidence levels — separate real gains from noise.
Google or a one-time email code. No password.