Problem
Analytics teams put a "talk to your data" agent in front of business users and find it answers every question, including those it cannot answer. Without table descriptions, metric definitions and explicit scope, the agent guesses columns and joins and returns wrong numbers with full confidence. One visible error costs the trust of all users, and the speaker says lost trust does not come back. Unit tests do not help, because answers are probabilistic. The talk uses Databricks Genie, but the practices fit any text-to-SQL agent.
Example from the article: in a demo, an agent with no metadata answered "sales revenue by product" with guessed columns, then admitted it could not say what Val1 meant.
Solution in brief
Analysts engineer trust upstream. They write instructions that say when the agent must not answer. They build a benchmark with ground-truth answers, including "no data" questions, to map its scope. They document the data and move business logic into a semantic layer, so the agent retrieves numbers instead of computing them.
Questions this case answers
- How to make a text-to-SQL analytics agent refuse questions it cannot answer and return correct numbers, verified by a benchmark with ground-truth answers
- How to decide which analytics experience to build, such as a dashboard, an app, an alert or an agent, for a recurring business question
- How to make a conversational analytics agent (text-to-SQL, a Genie space) trustworthy for business users
- How to evaluate an LLM data agent without unit tests, including questions it must refuse
Tools and methods
- Benchmarks — questions with analyst-written correct answers.
- Metric views — a semantic layer with joins and measures defined once.
- Validated generated metadata — table and column descriptions.
- Genie Workbench auto-optimize — an LLM-as-a-judge loop that tunes the agent against benchmarks.
- Context metrics — coverage, freshness, consumption, outcomes.
Google or a one-time email code. No password.