Problem
Risk teams at online retailers decline anomalous orders with rules, called strategies, that run in a decision engine. Writing them by hand depends on human review: an analyst cannot track every change in shopping behavior, and can inspect only about 50 of the roughly 1000 available features. Building a reasonable strategy for one anomaly takes a day, sometimes a week, while shopping patterns keep changing.
Example from the article: a bot attack on orders, stopped by one rule that combines four conditions on name mismatch, IP, address age and device age.
Solution in brief
Train a decision tree on labeled orders. Each branch is already a readable rule, so it can be deployed directly. Rank branches by the PRS score, a weighted harmonic mean of precision, recall and stability, pick the tree by AUC, and refine the next round with feedback from human review of false positives and false negatives.
Questions this case answers
- How to generate interpretable decision rules for adversarial events automatically from labeled history and keep them accurate as behavior changes
- How to generate fraud and anomaly rules for orders automatically instead of writing them by hand?
- How to choose the best decision-tree branch as a deployable rule when precision, coverage and stability all matter?
- How to keep automatically generated rules current as shopping patterns shift?
Tools and methods
- Decision tree — finds non-linear feature combinations and prints them as rules
- PRS score — ranks candidate rules by precision, recall (coverage) and stability together
- AUC on PRS-ranked leaves — chooses the best tree on a validation set
- Bootstrap and permutation of training data — measures how stable a rule is
- Feedback loop with human review — reweights falsely decided orders in the next round
Google or a one-time email code. No password.