Problem
Card testing is a fraud pattern where attackers check which stolen card numbers still work. It hides in normal traffic and is a small share of all transactions, so precise detection is hard. Tactics keep changing, so a model trained once goes stale. Unlike disputes or declines, card testing leaves no explicit label for training or for measuring prevalence and quality.
Example from the article: a spike in transactions at one business could be an attack or a flash sale, and the system has to tell them apart.
Solution in brief
Three models (overall prevalence, attack location, single transaction) jointly set a block threshold that moves with the attack. A loop turns each detected attack into derived labels and new features within hours, then retrains, tests offline and blue-green, and redeploys. A large transaction foundation model covers novel attacks the small models miss.
Questions this case answers
- How to keep an adversarial-behavior classifier current when positives have no ground-truth label and tactics shift within days
- How to set a blocking threshold that tightens during an attack and relaxes when it ends, so the rare positive class is caught with a low false positive rate
- How to keep a fraud model current when attackers keep changing tactics?
- How to get training labels for an attack type that leaves no explicit label such as a dispute or a decline?
Tools and methods
- Three-level scoring — prevalence model, location model (business, issuer, surface) and transaction model feed one dynamic threshold
- Derived labels — combine new attack intelligence, automated discovery from weak signals and expert review
- Shepherd (feature platform, built with Airbnb) — build new features with minimal code changes
- Flyte (ML orchestration) — standardized, automated retrain and evaluation workflows
- Blue-green test — run the old and new model side by side before full deployment
- Transaction foundation model — transformer trained on billions of transactions; embeddings feed attack classifiers
Google or a one-time email code. No password.