Problem
Optus, an Australian telecom, wanted business users to ask Genie (Databricks' natural-language data assistant) plain-English questions about its data. Genie needs table and column descriptions as context, and the 15,000 tables left by 30 years of migrations lacked them. Writing them by hand was estimated at about 6,000 hours. The built-in Databricks generator worked one table at a time and had no API.
Example from the article: the same column repeats across bronze, silver and gold tables, so per-table generation gives it a different description each time, and a flag like PSTP_ACT_flag cannot be decoded without Optus-specific knowledge.
Solution in brief
Inventory all columns from the information schema, deduplicate them by name, and generate each description once with an LLM prompt carrying a company glossary and optional sampled values. A reviewer approves each description in an app, and the approved text is applied to every table using the column. The open-source Databricks Metadata Generator implements this extract, generate, approve, apply flow.
Questions this case answers
- How to generate descriptions for thousands of tables in bulk by generating each distinct column once with a glossary-guided LLM prompt and then approving it through a review step
- How to keep AI-generated column descriptions consistent when the same column repeats across raw, cleaned and aggregated layers
- How to generate column descriptions in bulk for thousands of tables so a text-to-SQL assistant can answer business questions
- How to keep AI-generated column descriptions consistent when the same column repeats across bronze, silver and gold tables
Tools and methods
- Databricks Metadata Generator — open-source app implementing the flow
- Unity Catalog information schema — column inventory and target for applied comments
- Column deduplication and fingerprinting — one description per distinct column, lower LLM cost
- Prompt templates, data sampling, abbreviation glossary — context for the LLM
- Databricks Apps, Lakebase, Unity AI Gateway — review UI, audit store, model access
Google or a one-time email code. No password.
