Ship to Production, Darkly: Moving Fast, Staying Safe with ML Deployments

Bob Nugman8 Mar 2022

Problem

DoorDash fraud models run on nearly every interaction, with hundreds of features computed in real time. A new model version can be worse at stopping fraud, can block every transaction because of a defect, or can put heavy load on systems through expensive queries. Offline tests and backtests do not cover these cases: the final check of correctness happens only in production, on real and novel data. Models are also hard to cover with automated tests and hard to inspect.

Example from the article: a new fraud model that passes backtesting but blocks every attempted transaction, or one that makes such expensive queries that systems slow to a halt.

Solution in brief

Treat a model release like a dark code release. The model runs in production on shadow traffic that has no business effect, first at a fraction of a percent and then at 100%. Then it competes with the current champion in an A/B experiment and gets live traffic once it wins.

Questions this case answers
  • How to release a new decision model into a high-volume production system so that defects in correctness, features or load show up before the model affects any business decision
  • How to verify a replacement model against the incumbent on real traffic when offline tests cannot guarantee correctness
  • How to ship a new fraud-detection model to production without risking blocked transactions or overloaded systems?
  • How to check that inference-time features match training-time features before a model makes live decisions?

Tools and methods

  • Shadow traffic — the model is called on real requests, but its decisions are not used.
  • Rule engine and ML service — standard path to call the model with fault isolation, logging and metrics.
  • Feature-consistency check — specialized tooling that compares inference-time and training-time feature values.
  • Observability stack — dashboards, logs and alerting for system metrics.
  • Curie experimentation system — A/B comparison of the incumbent and the new model.
Continue reading — it's free

Google or a one-time email code. No password.