Problem
A team runs an LLM agent, for example a customer support chatbot, and collects traces for every call. The traces serve only to debug single failures in a trace viewer: search by request ID, open one trace. Nobody can say what the agent is asked, what each question type costs, or whether it reduces support load, so investment decisions rest on intuition.
Example from the article: a support agent for a product catalog also receives questions like "What is 25 times 17?", password resets and translation requests. Per-trace debugging never shows how large this off-pattern share is.
Solution in brief
Store agent traces as a queryable table and analyze them with SQL. Classify each query by intent with an LLM function, aggregate cost and LLM calls per intent, then add a customer_id argument to the agent function and join traces to customer and support-ticket tables. The result is agent KPIs next to business KPIs in one query.
Questions this case answers
- How to measure the business value of an LLM agent by joining its execution traces to customer and support-ticket data and computing cost per request type
- How to discover what users actually ask an agent, including requests it was never built for, from logged execution traces
- How to measure the business value of an LLM agent by joining its traces with customer and support-ticket data
- How to find what users actually ask an agent, including questions it was never meant to answer
Tools and methods
- MLflow autolog — captures traces, function arguments and LLM cost automatically.
- OpenTelemetry (OTel) spans in a Unity Catalog Delta table — makes traces queryable with SQL.
ai_classify— a SQL AI function that labels each root-span user query with an intent.variant_get— reads fields such as cost, span type and inputs from theattributescolumn.- JOIN to customer and ticket tables — links agent behavior to customer segments and escalations.
Google or a one-time email code. No password.