Problem
Grab's real-time fraud detection runs on a rule engine. Before a rule goes live, analysts need to know how it would have treated past events: how many it hits and whether it catches known false positives. Each team measured this its own way: Presto tables (slow, inaccurate), offline rebuilds of the engine payload (long setup) or shadow mode (days of waiting). One team kept a second rule engine only for backtesting, which doubled the work.
Example from the article: a rule scores a transaction (transaction_id, user_id, currency, amount, timestamp) and the engine returns Approve or Decline. Analysts want the total transactions, distinct users and amount by date, treatment and currency over the last couple of weeks.
Solution in brief
Grab stores every rule-engine event in time-partitioned files in S3. In the rule editor an analyst picks a rule and a time range, and a Spark job applies the rule to the stored events and returns a report.
Questions this case answers
- How to estimate the effect of a proposed decision rule on past events without running it on live traffic for days
- How to give non-engineer analysts a self-service way to run heavy batch simulations over logged decision events
- How to check what a new or changed fraud rule would have done on past events before it goes live?
- How to replace shadow-mode runs that take days with an offline replay of historical rule-engine events?
Tools and methods
- Rule backtesting (replay) — apply a rule to logged historical events and count hits per dimension
- Apache Kafka, Amazon S3 — event stream and time-partitioned storage
- Apache Spark on Amazon EMR — runs the rule script over stored events and scales with data size
- Amazon SQS, AWS Lambda — carry the job request and status between UI and Spark
- Kubernetes stream pipeline — saves the event stream to a separate AWS sub-account
Google or a one-time email code. No password.