Problem
A company puts an LLM assistant on top of its data warehouse, and users ask in business language while the schema speaks in column names. The assistant picks the literal reading and returns a plausible but wrong table. Marketing and sales users cannot tell the difference and may send wrong numbers to clients. The analytics team meanwhile cannot write a campaign takeaway for every report.
Example from the article: at Omni, "how much business have we closed this quarter?" returned both Closed Won and Closed Lost Salesforce opportunities. The user meant "won", but a column called is_closed matched the word "closed". The result overstated everything.
Solution in brief
Put a layer of written business context (ai_context) on top of the semantic layer: definitions, synonyms, fiscal-calendar rules and the way to write an insight. Then run a loop: ask real questions, log the answers, have humans grade them, revise the context and test again. Condé Nast used this to generate campaign takeaways inside its dashboards.
Questions this case answers
- How to make an LLM analytics assistant choose the intended reading of an ambiguous business term over a literal column match
- How to run a human-graded feedback loop on a natural-language analytics assistant after launch and keep it accurate
- How to make an LLM analytics assistant return correct answers for ambiguous business terms like 'closed'
- How to write business context (ai_context) so an AI assistant on top of a semantic layer follows company definitions
Tools and methods
- Semantic layer (with dbt) — metric definitions, joins and security rules the AI works from
ai_context— free-text instructions attached to a model or topic: definitions, synonyms, filters, output format- Human grading loop — ask, log, grade, revise context, test again
- Question-log mining — Slack channels, emails and assistant logs show what users really ask
- Databricks, dbt, ChatGPT — warehouse and pipeline; ChatGPT refined the written context
Google or a one-time email code. No password.