Problem
A fintech company screens person-to-person transfers with rules. Some block a transaction, most pass it, others hold it for analysts, who call the customer. Maintaining the rules gets more expensive, and they do not adapt to new fraud schemes. The business wants an ML model under hard limits: scoring within one second, interpretable decisions, a limited analyst headcount, few blocked good transactions. The team must first agree what "better" means in money and how to prove it.
Example from the article: two teams answered this case in a one-day design doc contest, sizing the pilot for 100,000 transactions per day.
Solution in brief
Write the design doc before modeling. Define success as one profit formula in rubles (penalties, complaint-driven churn, analyst cost) and validate it in a 50/50 pilot. Put the model behind the existing rules, split its output into allow, send to analysts and block with two thresholds, and keep a switch back to the old system.
Questions this case answers
- How to define a money-denominated business metric for a model that replaces rule-based screening, and agree on it before modeling
- How to set allow, review and block thresholds for a screening model with a limited review team, and validate them in a pilot when penalties arrive with a delay
- How to define a business metric in money terms for an ML anti-fraud system that replaces rule-based checks
- How to design a pilot A/B test for a fraud model when penalties arrive with a delay
Tools and methods
- Profit formula with normalization — business metric in rubles, divided by the maximum possible profit on the same transactions.
- Mann-Whitney test — compares penalty amounts between pilot arms.
- Three-outcome thresholding — allow, send to analysts, block.
- Boosting plus logistic regression — candidate model with interpretable baseline.
- Isolation forest — surfaces unlabeled fraud among outliers.
- Kafka, Spark Streaming, Airflow, Prometheus, Grafana — serving, nightly retraining and monitoring stack.
Google or a one-time email code. No password.