Practical Text-to-SQL for Data Analytics

Albert Chen, Manas Bundele, Gaurav Ahlawat, Patrick Stetz, Zhitao W., Qiang Fei, Don Jung, Audrey Chu, Bharadwaj Jayaraman, Ayushi Panth, Yatin Arora, Sourav Jain, Renjith Varma, Alex Ilin, Iuliia Melnychuk, Chelsea C., Joyan Sil, Xiaofeng Wang9 Dec 2024

Problem

Data experts at a large tech company spend much of their time pointing colleagues to the right data. A text-to-SQL proof of concept is easy to build, but a real warehouse has millions of tables, many undocumented or deprecated, and one question can mean different things to different teams. LinkedIn built SQL Bot so employees outside the data team can query data themselves.

Example from the article: the question "What was the average CTR yesterday?" needs a different table depending on whether the asker works on email notifications, ads or search quality.

Solution in brief

Treat text-to-SQL as retrieval first and generation second. Narrow tables by popularity and user context, re-rank them with LLMs over a knowledge graph, write the query step by step, and validate it with checks that use information the writer lacked. Measure on an expert-built benchmark.

Questions this case answers
  • How to make an LLM answer questions over a very large, poorly documented data warehouse with the correct data source
  • How to measure the accuracy of generated queries when one question has several valid answers
  • How to make a text-to-SQL assistant pick the right tables in a warehouse with millions of tables?
  • How to answer an ambiguous question such as average CTR when it means different things in different teams?

Tools and methods

  • Embedding-based retrieval (EBR) — finds tables matching the question.
  • Independent Component Analysis (ICA) — turns user-dataset access history into use-case components for personalization.
  • Knowledge graph — links users, tables and fields with catalog, log and user-supplied attributes.
  • LLM re-rankers — pick 7 tables out of 20, then the fields.
  • Plan-and-solve writer, self-correction agent — build the query stepwise and fix errors.
  • LLM-as-a-judge with expert review — scores query accuracy.
  • LangChain, LangGraph, DataHub — the multi-agent framework and the metadata catalog.
Continue reading — it's free

Google or a one-time email code. No password.