BI for AI: Transforming OTel Agent Telemetry into Enterprise Analytics

Oleksandra Bovkun15 Jun 2026

Problem

A team runs an LLM agent, for example a customer support chatbot, and collects traces for every call. The traces serve only to debug single failures in a trace viewer: search by request ID, open one trace. Nobody can say what the agent is asked, what each question type costs, or whether it reduces support load, so investment decisions rest on intuition.

Example from the article: a support agent for a product catalog also receives questions like "What is 25 times 17?", password resets and translation requests. Per-trace debugging never shows how large this off-pattern share is.

Solution in brief

Store agent traces as a queryable table and analyze them with SQL. Classify each query by intent with an LLM function, aggregate cost and LLM calls per intent, then add a customer_id argument to the agent function and join traces to customer and support-ticket tables. The result is agent KPIs next to business KPIs in one query.

Questions this case answers
  • How to measure the business value of an LLM agent by joining its execution traces to customer and support-ticket data and computing cost per request type
  • How to discover what users actually ask an agent, including requests it was never built for, from logged execution traces
  • How to measure the business value of an LLM agent by joining its traces with customer and support-ticket data
  • How to find what users actually ask an agent, including questions it was never meant to answer

Tools and methods

  • MLflow autolog — captures traces, function arguments and LLM cost automatically.
  • OpenTelemetry (OTel) spans in a Unity Catalog Delta table — makes traces queryable with SQL.
  • ai_classify — a SQL AI function that labels each root-span user query with an intent.
  • variant_get — reads fields such as cost, span type and inputs from the attributes column.
  • JOIN to customer and ticket tables — links agent behavior to customer segments and escalations.
Continue reading — it's free

Google or a one-time email code. No password.