Problem
Scammers use the BlaBlaCar carpooling marketplace to obtain payment details. They create fake profiles with no ratings and low prices, or contact members outside the platform. The team had no agreed measure of the fraud problem, and the direct signal, victim reports, arrives up to 3 weeks late. Small fixes against fake accounts were tempting, but without a metric they could not be judged. The team wanted to know first whether machine learning (ML) was worth the investment.
Example from the article: a rule flagged trips published on the hour (6:00, 7:00), where fraudsters' scripts ran. The fraudsters moved their schedule, and the rule then mostly caught legitimate users. The article calls the example made-up.
Solution in brief
Set one company-wide KPI and several daily proxy metrics, then start with manual review of the riskiest cohorts and a few expert rules. A tool that tracks each rule's performance shows when rules decay and gives sponsors evidence for an ML system.
Questions this case answers
- How to define a lagging outcome metric and fast daily proxy metrics for an adversarial abuse problem before choosing a detection method
- How to decide between hand-written rules and a learned model when adversaries change behavior faster than the detector is updated
- How to define an anti-fraud KPI and daily proxy metrics before building any detection model?
- How to decide whether a fraud problem is worth a machine learning system or can be handled by manual work and rules?
Tools and methods
- Victims per active members KPI — quarterly per-country objectives from scam reports
- Proxy metrics from manual labeling — daily share of bookings made on scammers' trips, later share of fraudulent supply
- Manual review of high-risk cohorts — stops active scams and collects labels
- Expert business rules — first version of the detector
- Rule performance dashboard — shows each rule's performance over time
Google or a one-time email code. No password.
