Problem
Advertising revenue at LINE Ads moves every day. KPI dashboards show how much revenue moved but not which products, industries or advertisers moved it, so business teams compared breakdowns by hand. A single prompt with the day's aggregates gave unstable day-over-day (DoD), month-over-month (MoM) and year-over-year (YoY) numbers, skipped high-revenue contributors, broke formatting rules and could not link a change to its drivers.
Example from the article: a rise in revenue of an Official Account (OA) ad product group is traced to detailed product, industry, then advertiser, and the path differs from day to day.
Solution in brief
Code does all arithmetic and picks candidate items. A ReAct agent with five tools and a bounded exploration graph looks for causes, and rules keep only consistent findings. The LLM writes each section from only the context it needs, and every run is traced and scored against a rubric.
Questions this case answers
- How to produce a recurring metric-change report with an LLM when the LLM cannot be trusted with arithmetic or with choosing which contributors to cover
- How to let an LLM agent drill down through dimensions to explain a metric change while bounding its freedom to explore
- How to generate a daily KPI change report with an LLM without wrong arithmetic or missing top contributors?
- How to let an LLM agent find the root cause of a revenue change without letting it query anything it likes?
Tools and methods
- Pre-aggregated KPI tables (Spark ETL) — DoD/MoM/YoY changes and top movers
- ReAct agent with tool calling — picks the next drill-down step until a cause is found
- Trino — ad-hoc queries for advertiser history and trends
- Exploration graph — limits which dimensions the agent may move between
- Evidence grading — labels findings
driver_eligible,caution_onlyormonitoring_signal - Langfuse and LLM-as-a-judge — traces, rubric scoring and a human annotation queue
Google or a one-time email code. No password.
