Problem
A team runs several ML services in production and learns about failures from the customer. Code tests do not catch the cases that hurt: input data that differs from what the model expects, or a new client whose data looks unlike the training set. The service then returns a wrong answer or an error, and developers lose time finding which request failed and why.
Example from the article: Celsus runs five medical-imaging ML services (mammography, fluorography, chest X-ray, chest CT, brain CT). A service expecting a lung X-ray received a foot and "found" a pneumothorax on it. In another case, images came in one color format for a month and then arrived inverted.
Solution in brief
Treat an incident as five stages (trigger, start, detection, root cause analysis, resolution) and cover each with a cheap tool. Sentry alerts in Slack give detection. A small in-house web app finds the request by ID and shows period statistics. ELK, the Kubernetes dashboard, Grafana and Prometheus explain the cause.
Questions this case answers
- How to detect silent failures of production model services, where input data differs from what the model expects, before the affected users report them
- How to trace why a model service returned a wrong result or ran slowly, from requests to individual processing steps
- How to find out about an incident in a production ML service before the customer does
- How to investigate why an ML service returned a wrong result or ran slowly
Tools and methods
- Sentry + Slack bot — error alerts
- Stream-tv (Streamlit, FastAPI, PostgreSQL) — in-house app: search by ID, period statistics
- ELK (Elasticsearch, Logstash, Kibana) — logs and intermediate model features
- Kubernetes dashboard — which component or pod is down, pod logs
- Grafana — CPU, memory, disk, request counts per pod
- Prometheus — execution time of individual pipeline steps
- Evidently AI — data drift monitoring (planned, not implemented)
Google or a one-time email code. No password.
