Problem
A team at LY Corporation connected Claude to its AI Assistant service database through a database MCP server, expecting it to democratize data analysis across the organization. The generated SQL looked fine but was wrong on meaning. Asked for this month's DAU, the model chose the wrong table among dozens containing "user". It also did not know that "active" counts only certain API endpoints. It tried to drop tables as "schema cleanup" and returned cryptic syntax errors. It also raised security flags as soon as production data came up.
Example from the article: "What's this month's DAU?" returned a confident query on a wrong user table.
Solution in brief
Put business knowledge into human-editable YAML and CSV files that an MCP server gives to the model, and force every question through a fixed seven-step tool workflow with a validation gate. Confirmed question-SQL pairs are added to a CSV of golden examples and automatically expanded into variations.
Questions this case answers
- How to make an LLM answer business-metric questions over a warehouse with SQL that follows the organization's own metric definitions and passes validation before delivery
- How to keep a text-to-SQL system improving from user-confirmed answers while keeping the training examples reviewable by analysts
- How to make an LLM pick the right tables and use our own definition of a metric such as DAU?
- How to stop an LLM connected to a database from running destructive or unvalidated SQL?
Tools and methods
- Domain-specific MCP server — exposes one tool per workflow step to the LLM client
- YAML configuration files — metric definitions, SQL templates and segmentation rules edited by domain experts
- CSV golden dataset — confirmed question-SQL pairs, used for similarity lookup and reviewable by analysts
- Validation pipeline — syntax, security and performance checks before SQL reaches the user
- Claude — the LLM client that follows the workflow from the system prompt
Google or a one-time email code. No password.
