From Eight Hours to Eight Minutes: Automating A/B Test Analysis on Databricks

Joel Dias, Sanjay Ashok18 Jun 2026

Problem

SEGA HARDlight is a small mobile studio running live games (Sonic Dash, Sonic Forces) for about a million players a day. Product wants ever more experiments on features, onboarding and the game economy, and a handful of analysts must analyse them all. One experiment took 8+ hours of analyst time: SQL extracts, a Python script on a local machine, manual reading of outputs, a report. Two analysts could apply different thresholds to the same result, so interpretations were inconsistent.

Example from the article: in a Sonic Forces test, ad revenue rose 19.21% (significant) but LTV did not change significantly, and the dashboard summary still recommended a rollout with LTV monitoring.

Solution in brief

Fix one statistical method per metric family and run it automatically on every test in a lakehouse pipeline. A daily job applies winsorization, sample-size gates and FDR correction, then publishes a layered dashboard with an LLM summary on top.

Questions this case answers
  • How to standardize and automate the analysis of many parallel experiments on conversion, retention and numeric metrics so that readouts do not depend on the analyst
  • How to estimate a long-term value effect with a confidence interval when the outcome accrues over time and is heavy-tailed
  • How to automate A/B test analysis when a small analytics team runs dozens of parallel experiments
  • How to analyse conversion, retention, numeric metrics and LTV in one standard experiment pipeline

Tools and methods

  • Chi-square plus Bayesian Beta — conversion to spender.
  • Kaplan-Meier plus log-rank test — retention as a full curve.
  • Welch's t-test — continuous metrics.
  • Exponential curve fit with Wald-style 95% CI — LTV projection.
  • Winsorization, bootstrap, sample-size gates, FDR correction — outliers and false positives.
  • Lakeflow, Unity Catalog, MLflow, AI/BI dashboards, Genie — pipelines, lineage, run versioning, dashboards.
Continue reading — it's free

Google or a one-time email code. No password.