Problem
SEGA HARDlight is a small mobile studio running live games (Sonic Dash, Sonic Forces) for about a million players a day. Product wants ever more experiments on features, onboarding and the game economy, and a handful of analysts must analyse them all. One experiment took 8+ hours of analyst time: SQL extracts, a Python script on a local machine, manual reading of outputs, a report. Two analysts could apply different thresholds to the same result, so interpretations were inconsistent.
Example from the article: in a Sonic Forces test, ad revenue rose 19.21% (significant) but LTV did not change significantly, and the dashboard summary still recommended a rollout with LTV monitoring.
Solution in brief
Fix one statistical method per metric family and run it automatically on every test in a lakehouse pipeline. A daily job applies winsorization, sample-size gates and FDR correction, then publishes a layered dashboard with an LLM summary on top.
Questions this case answers
- How to standardize and automate the analysis of many parallel experiments on conversion, retention and numeric metrics so that readouts do not depend on the analyst
- How to estimate a long-term value effect with a confidence interval when the outcome accrues over time and is heavy-tailed
- How to automate A/B test analysis when a small analytics team runs dozens of parallel experiments
- How to analyse conversion, retention, numeric metrics and LTV in one standard experiment pipeline
Tools and methods
- Chi-square plus Bayesian Beta — conversion to spender.
- Kaplan-Meier plus log-rank test — retention as a full curve.
- Welch's t-test — continuous metrics.
- Exponential curve fit with Wald-style 95% CI — LTV projection.
- Winsorization, bootstrap, sample-size gates, FDR correction — outliers and false positives.
- Lakeflow, Unity Catalog, MLflow, AI/BI dashboards, Genie — pipelines, lineage, run versioning, dashboards.
Google or a one-time email code. No password.