Real-time Fraud Detection with Yoda and ClickHouse

Nick Shieh, Shen Zhu, Xiaobing Xia18 Mar 2024

Problem

Instacart fights many kinds of fraud: fake accounts, payment fraud, and collusion between customers and shoppers. Patterns change often, and each new heuristic used to need engineers. Rules also have to decide in a fraction of a second on the freshest data, or the damage is already done. Engineers used to spend about 80% of their time on heuristic development.

Example from the article: a shopper's card swipe at the store is larger than the in-store checkout amount for the same order. A rule suspends the shopper if the swipe is at least X times the checkout amount.

Solution in brief

Instacart built Yoda, a decision service where analysts write rules in a UI. Rules use features that are defined as parameterized SQL in YAML and run against ClickHouse at request time. Postgres changes and app events reach ClickHouse through Kafka and a config-driven Flink job, so a new rule needs no engineering code once the tables exist.

Questions this case answers
  • How to let non-engineers define and deploy rules on fresh operational data when decisions must be made within a fraction of a second and fraud patterns keep changing
  • How to serve aggregated features computed on demand from streamed transactional data without a precomputed feature pipeline for every new rule
  • How to let fraud analysts ship new real-time fraud rules without engineers writing code for each one?
  • How to compute fresh features from operational data in milliseconds at decision time?

Tools and methods

  • Yoda — in-house decision platform: rules UI, feature system, evaluation service, action dispatch
  • ClickHouse — real-time columnar datastore that answers the feature queries
  • Debezium + Kafka Connect — change data capture from PostgreSQL into Kafka
  • Apache Kafka / Apache Flink — transport and a YAML-configured ingestion job into ClickHouse
  • YAML feature definitions — analyst-owned SQL features reusable across rules
Continue reading — it's free

Google or a one-time email code. No password.