Incidents in ML Services and How to Monitor Them

Mikhail Suslov25 May 2024

Problem

A team runs several ML services in production and learns about failures from the customer. Code tests do not catch the cases that hurt: input data that differs from what the model expects, or a new client whose data looks unlike the training set. The service then returns a wrong answer or an error, and developers lose time finding which request failed and why.

Example from the article: Celsus runs five medical-imaging ML services (mammography, fluorography, chest X-ray, chest CT, brain CT). A service expecting a lung X-ray received a foot and "found" a pneumothorax on it. In another case, images came in one color format for a month and then arrived inverted.

Solution in brief

Treat an incident as five stages (trigger, start, detection, root cause analysis, resolution) and cover each with a cheap tool. Sentry alerts in Slack give detection. A small in-house web app finds the request by ID and shows period statistics. ELK, the Kubernetes dashboard, Grafana and Prometheus explain the cause.

Questions this case answers
  • How to detect silent failures of production model services, where input data differs from what the model expects, before the affected users report them
  • How to trace why a model service returned a wrong result or ran slowly, from requests to individual processing steps
  • How to find out about an incident in a production ML service before the customer does
  • How to investigate why an ML service returned a wrong result or ran slowly

Tools and methods

  • Sentry + Slack bot — error alerts
  • Stream-tv (Streamlit, FastAPI, PostgreSQL) — in-house app: search by ID, period statistics
  • ELK (Elasticsearch, Logstash, Kibana) — logs and intermediate model features
  • Kubernetes dashboard — which component or pod is down, pod logs
  • Grafana — CPU, memory, disk, request counts per pod
  • Prometheus — execution time of individual pipeline steps
  • Evidently AI — data drift monitoring (planned, not implemented)
Continue reading — it's free

Google or a one-time email code. No password.