Full talk · edited transcript

Behind the Curtain: How We Do Eval in Genie

Shanshan Zheng17 Jun 2026

Shanshan Zheng, Databricks (Genie engineering team)

Evaluating AI systems is hard, especially when correctness is not binary. This talk describes how the Genie engineering team evaluates at scale: how "good" is defined, and how automated eval pipelines are built from offline benchmarks, human-in-the-loop validation and production feedback loops, along with the tradeoffs encountered.

Continue reading — it's free

Google or a one-time email code. No password.