Problem
Instacart fights many kinds of fraud: fake accounts, payment fraud, and collusion between customers and shoppers. Patterns change often, and each new heuristic used to need engineers. Rules also have to decide in a fraction of a second on the freshest data, or the damage is already done. Engineers used to spend about 80% of their time on heuristic development.
Example from the article: a shopper's card swipe at the store is larger than the in-store checkout amount for the same order. A rule suspends the shopper if the swipe is at least X times the checkout amount.
Solution in brief
Instacart built Yoda, a decision service where analysts write rules in a UI. Rules use features that are defined as parameterized SQL in YAML and run against ClickHouse at request time. Postgres changes and app events reach ClickHouse through Kafka and a config-driven Flink job, so a new rule needs no engineering code once the tables exist.
Questions this case answers
- How to let non-engineers define and deploy rules on fresh operational data when decisions must be made within a fraction of a second and fraud patterns keep changing
- How to serve aggregated features computed on demand from streamed transactional data without a precomputed feature pipeline for every new rule
- How to let fraud analysts ship new real-time fraud rules without engineers writing code for each one?
- How to compute fresh features from operational data in milliseconds at decision time?
Tools and methods
- Yoda — in-house decision platform: rules UI, feature system, evaluation service, action dispatch
- ClickHouse — real-time columnar datastore that answers the feature queries
- Debezium + Kafka Connect — change data capture from PostgreSQL into Kafka
- Apache Kafka / Apache Flink — transport and a YAML-configured ingestion job into ClickHouse
- YAML feature definitions — analyst-owned SQL features reusable across rules
Google or a one-time email code. No password.