Grab Experiment Decision Engine - a Unified Toolkit for Experimentation

Ruike Zhang9 Apr 2024

Problem

Grab's experimentation platform (GrabX) runs thousands of experimental variants a week. Experimenters analyzed them one by one with generic online calculators and ad hoc tests. These tools ignore marketplace specifics such as repeated observations per user or switchback designs. This led to poorly designed experiments, mismatched evaluation methods and long review cycles over disputed approaches. Tutorials rarely cover multi-sided marketplaces, so teams cannot fix this on their own.

Illustrative example: an analyst sizes a switchback test of a pricing change with an online calculator built for independent users, and later analyzes it with a plain t-test that ignores spillover between timeslices.

Solution in brief

Grab built the GrabX Decision Engine, a Python package open-sourced inside the company. It bundles a pre-experiment advisor, a post-experiment analysis toolbox and advanced tools such as variance reduction. Defaults depend on the design: between-subject (users are split) or within-subject (switchback).

Questions this case answers
  • How to make experiment design and analysis consistent across many teams when units produce repeated, correlated observations and treatments interfere across time
  • How to ship statistical defaults that match each randomization design so analysts stop using generic calculators and ad hoc tests
  • How to standardize experiment design and analysis across many teams on one experimentation platform?
  • How to size a between-subject experiment when each user produces many observations?

Tools and methods

  • Trusted Advisor — sample-size recommendation for between-subject tests, with clustered-error adjustment
  • Post-experiment toolbox — default tests, regressions and warnings for both designs
  • Variance reduction — stratified sampling, post-stratification, CUPED, MLRATE
  • Multiple-comparison corrections — FWER and FDR procedures
  • Experiment Council and GrabCausal Methodology Bank — people and a shared workspace around the package
Continue reading — it's free

Google or a one-time email code. No password.