SafeChat: DoorDash's AI-powered safety feature

Bruna Pereira30 Oct 2025

Problem

DoorDash handles millions of chat messages and hundreds of thousands of images and calls every day between couriers (Dashers) and customers. A small share is abusive, threatening or sexual and must be stopped before delivery. An LLM on every message is too slow and costly, and an off-the-shelf image API gave too many false positives.

Example from the article: an abusive customer message scores 4 or higher (scale 0 to 5) in two categories, so the Dasher may cancel the order.

Solution in brief

Text goes through a cascade: a cheap high-recall filter clears most traffic, and only the rest reaches a precise LLM that scores profanity, threats and sexual content. After months of traffic, an internal model trained on the collected data replaced the first layers. Images and voice went through shadow tests and human review before any automatic action.

Questions this case answers
  • How to screen a high-volume stream of user-generated text for abuse in real time under a tight cost budget
  • How to move a new automated moderation model from passive monitoring to automatic enforcement while keeping errors under control
  • How to moderate millions of chat messages in real time without paying for an LLM on every message?
  • How to roll out image moderation when an off-the-shelf vision API gives too many false positives?

Tools and methods

  • LLM cascade — cheap filter, fast LLM, then precise LLM, so costly calls see little traffic
  • Knowledge distillation — an internal model trained on about 10 million cascade data points
  • Shadow mode — run a candidate on all traffic without acting on it
  • Human in the loop — reviewers check flagged items to tune categories and thresholds
  • Threshold selection — per-category thresholds balance precision and recall
  • Text classification, image classification — the underlying scoring tasks
Continue reading — it's free

Google or a one-time email code. No password.