ML Observability: Bringing Transparency to Payments and Beyond

Tanya Tang, Andrew Mehrmann18 Aug 2025

Problem

Netflix routes thousands of payment transactions per minute to payment processors, and an ML model increasingly makes that choice. Before ML, stakeholders could read the static routing rules to see why a route got its volume. A model hides this, and technical metrics such as precision and recall do not answer their questions: how much traffic went down this route, and where ML underperforms. Without explanations, stakeholders cannot discuss routing with banks and processors or trust the model.

Example from the article: an alert showed that the model consistently cut traffic to one route every Tuesday, and the team needed to know why.

Solution in brief

Build three linked modules around the model: logging, monitoring and explaining. Log every decision with a fixed schema, monitor simple outcome metrics on top of those logs, and use SHAP to explain decisions for business groups and for the gap between similar candidate routes.

Questions this case answers
  • How to make the decisions of a multi-layer ML policy explainable to non-technical stakeholders at the level of business groups, not single events
  • How to monitor a production decision model with outcome-focused metrics when technical model metrics do not answer stakeholders' questions
  • How to explain to business stakeholders why an ML routing model sends traffic down a given route?
  • How to monitor an ML decision system with outcome metrics instead of precision and recall?

Tools and methods

  • Decision logging schema — one set of logged entities for records, features, model, score, cutoffs and timestamps
  • Online outcome metrics — "business analytics on your models", such as traffic share per route and region
  • SHAP — feature attributions per decision, aggregated by any group stakeholders pick
  • Layer plot for bandit policies — shows how penalty and guardrail layers change the model's own traffic split
Continue reading — it's free

Google or a one-time email code. No password.