Problem
Dashboards are green (CPU, latency, servers, database) while one group of customers quietly fails. The speaker calls this a gray failure: it is partial and grows as more customers hit it. Teams usually learn about it from support tickets, which is slow, and many affected customers never file one. At Databricks it shows up as a spike in user errors: many users fail to start one cluster type in one region with INVALID_ARGUMENT, an error that looks like the user's fault.
Example from the article: a deploy at 09:30 breaks credit card checkout for 1 in 20 customers. The first complaint comes at 12:40, support escalates at 14:25, and the fix ships at 16:00.
Solution in brief
RADAR (Reliability Anomaly Detection, Alerting, and Root Cause Analysis) runs unsupervised streaming anomaly detection on error counts split by dimensions. Alerts are enriched, filtered by significance, deduplicated and filed as a ticket to the owning team with a Genie dashboard link. A scaffold file lets an AI agent build the same pipeline for any metric.
Questions this case answers
- How to detect a silent partial failure that affects one slice of users and does not show in aggregate health metrics
- How to alert on spikes in error counts across many dimension slices without hand-tuned thresholds
- How to detect a silent partial failure that dashboards do not show before customers complain
- How to alert on a spike in user errors without hand-tuned thresholds
Tools and methods
- SPOT model — streaming anomaly detection per series, 14-day baseline
- Significance filtering and deduplication — cut alert noise before a ticket is filed
- Databricks stack — ingestion, storage, model serving, SQL Alerts, Genie (LLM assistant), dashboards
- Databricks Asset Bundle — deploys all components together
- scaffold.md plus prompt — recipe that an LLM agent follows to build RADAR
Google or a one-time email code. No password.