Building a multi-agent pipeline for NL-to-SQL analytics

Shinae Choi5 Jan 2026

Problem

The LY Corporation team built an NL-to-SQL system for user-behavior event logs. Its first version ran one flow: interpret the question, build an intent, write SQL, explain the result. When an answer was wrong, nobody could say which step failed, and one early error spread through the whole result. Every new rule made the prompt longer and more likely to conflict with old rules.

Example from the article: the question "How many AU yesterday?" must always mean "users who sent a message or selected an AI recommendation" and always produce the same query, however the question is worded.

Solution in brief

Pre-compute the business logic in ETL. The LLM then only chooses a column and never rebuilds a metric from raw JSON events. Then split the work into five agents (Dialogue, Interpret, Precheck, Execute, Present) that pass structured JSON. Each agent gets only the context it needs, and each stage fails visibly on its own.

Questions this case answers
  • How to make a natural-language analytics assistant over denormalized event logs produce consistent, auditable queries when one prompt carrying all rules becomes unmanageable
  • How to let an LLM answer metric questions without writing the metric logic itself each time
  • How to make an NL-to-SQL assistant return the same SQL for differently worded questions about one metric?
  • How to find which step of an NL-to-SQL pipeline caused a wrong answer?

Tools and methods

  • Action flags and user-day data marts — ETL turns raw events into boolean flags, aggregates them per day and user, and publishes one mart per analytics domain.
  • Metric ontology — maps synonyms of a metric to one key, its definition, table and filter logic.
  • Intent JSON — structured contract between the Interpret and Precheck agents.
  • SQL template registry and guardrails — Precheck fills a fixed template per metric after safety checks.
Continue reading — it's free

Google or a one-time email code. No password.