Automated Experiment Analysis - Making experimental analysis scalable

Albert Cheng30 May 2022

Problem

Before the automated analysis, every experiment on Grab's experimentation platform (GrabX) was analysed by hand, case by case. Analysts rebuilt the same data pipelines and basic comparisons each time, so deeper analyses got little time. Results were not standardised, so they could be wrong or late, and the platform could not enforce data governance. Teams with different skills used different techniques, and advanced ones did not spread without training.

Example from the article: some teams used more advanced techniques that sped up experiments at lower resource cost, but other teams could not reuse them without considerable training.

Solution in brief

Grab extended GrabX with a data product that links experiment setup to results. Users choose success metrics when they create an experiment, a daily pipeline computes those metrics per experiment subject, and an internal Python package runs the standard tests and publishes results to an embedded dashboard. Metric definitions, data preparation and statistics are separate components.

Questions this case answers
  • How to standardize and automate post-experiment analysis when every team computes metrics and tests in its own ad-hoc way
  • How to decouple metric definitions, data preparation and statistical tests so each can be improved on its own
  • How to stop analysts from rebuilding the same data pipeline and basic A/B analysis for every experiment?
  • How to make post-experiment results reproducible and comparable across teams under one company standard?

Tools and methods

  • Metrics glossary — catalogue of metric definitions behind an API, so users pick metrics instead of coding them.
  • Bronze, silver, gold datasets — three pipeline layers: configs, computed per-subject metrics, final results.
  • Decision Engine — internal Python package that runs the statistical tests on silver data.
  • Azure Data Factory, Databricks, Apache Spark — scheduling and running the ETL jobs.
  • Power BI — results dashboard embedded in the GrabX experiment page.
Continue reading — it's free

Google or a one-time email code. No password.