How DoorDash Upgraded a Heuristic with ML to Save Thousands of Canceled Orders

Slava Nikitin, Gabriel Lerner10 Jan 2023

Problem

DoorDash does not always know whether a store is really open, because stores run independently and their hours go stale. When a courier (Dasher) arrives and finds the store closed, they file a "Dasher reports store closed" (DRSC) report with a photo. Each report triggers costly actions: cancel and refund the order, or pause the store. A few reports are wrong, and acting on them cancels orders that could have been delivered or pauses open merchants. Thousands of reports arrive daily, so manual review is too slow and expensive, and fixed rules are fast but only moderately accurate.

Example from the article: a Dasher reports a closed store, yet another Dasher completes a pickup there within 15 minutes of the report, so the store was probably open.

Solution in brief

Estimate the probability that the store is closed given the report. A single LightGBM model combines store and Dasher history with a score from an image classifier on the storefront photo. Thresholds on the probability pick the action: reassign the order, cancel it, or cancel it and pause the store.

Questions this case answers
  • How to validate unreliable user-submitted reports at scale when the true state of the world is never directly observed
  • How to derive training labels from downstream behavior when direct verification of the target is prohibitively expensive
  • How to map a predicted probability to one of several costly actions using thresholds
  • How to decide whether to trust a user-submitted report when the ground truth is never observed?

Tools and methods

  • LightGBM — one model that outputs the probability the store is closed
  • Image classifier on the storefront photo — compresses the picture into one number used as a feature
  • Proxy labels from follow-up behavior — substitute for the unknown true store status
  • Probability thresholds — map the 0–1 probability to three actions
Continue reading — it's free

Google or a one-time email code. No password.