The ML flywheel: How we continually improve our models to reduce card testing

Ryan Drapeau, Inayat Khosla12 Dec 2024

Problem

Card testing is a fraud pattern where attackers check which stolen card numbers still work. It hides in normal traffic and is a small share of all transactions, so precise detection is hard. Tactics keep changing, so a model trained once goes stale. Unlike disputes or declines, card testing leaves no explicit label for training or for measuring prevalence and quality.

Example from the article: a spike in transactions at one business could be an attack or a flash sale, and the system has to tell them apart.

Solution in brief

Three models (overall prevalence, attack location, single transaction) jointly set a block threshold that moves with the attack. A loop turns each detected attack into derived labels and new features within hours, then retrains, tests offline and blue-green, and redeploys. A large transaction foundation model covers novel attacks the small models miss.

Questions this case answers
  • How to keep an adversarial-behavior classifier current when positives have no ground-truth label and tactics shift within days
  • How to set a blocking threshold that tightens during an attack and relaxes when it ends, so the rare positive class is caught with a low false positive rate
  • How to keep a fraud model current when attackers keep changing tactics?
  • How to get training labels for an attack type that leaves no explicit label such as a dispute or a decline?

Tools and methods

  • Three-level scoring — prevalence model, location model (business, issuer, surface) and transaction model feed one dynamic threshold
  • Derived labels — combine new attack intelligence, automated discovery from weak signals and expert review
  • Shepherd (feature platform, built with Airbnb) — build new features with minimal code changes
  • Flyte (ML orchestration) — standardized, automated retrain and evaluation workflows
  • Blue-green test — run the old and new model side by side before full deployment
  • Transaction foundation model — transformer trained on billions of transactions; embeddings feed attack classifiers
Continue reading — it's free

Google or a one-time email code. No password.