How we evaluated the business impact of marketing campaigns

Jie Zhang23 May 2024

Problem

Grab sends millions of marketing messages a day (emails, pushes, feed items) but initially could not say whether they raised transactions or bookings. Campaign managers had to ask marketing analysts for each assessment. That does not scale with the number of campaigns, and different analysts measure impact slightly differently, so results drift over time. Without a trusted number, managers cannot cut wasteful campaigns that degrade the user experience.

Example from the article: a food campaign is judged by food GMV per user in the treatment and hold-out groups, not by ride bookings.

Solution in brief

Grab uses two tools. A marketing attribution model (MAM) splits credit among touchpoints and compares different campaigns. For one campaign, a hold-out group that receives no message is compared with the treated group on per-user GMV and revenue, with stratified groups, a significance test and metrics narrowed to the campaign's business line.

Questions this case answers
  • How to estimate the causal effect of a single outbound communication on a business metric when the control group has no attributed events and only aggregate totals can be compared
  • How to automate campaign impact reporting so that results stay consistent across analysts and scale with the number of campaigns
  • How to measure whether a single marketing campaign (push, email, feed message) changed GMV or revenue?
  • How to give campaign managers self-service impact analysis instead of ad-hoc analyst requests?

Tools and methods

  • Marketing attribution model (MAM) — credits each touchpoint by relevance and recency; compares campaigns, not a single one
  • Hold-out group — users in the target audience who get no message; the control
  • Stratified split — keeps the same user distribution in both groups
  • Statistical significance test (CLT-based) — rules out random differences between groups
  • jStat — JavaScript library that computes the significance in Grab's report
Continue reading — it's free

Google or a one-time email code. No password.