In-house monitoring of Payments success rate

Prabodh Swain17 Feb 2022

Problem

Blinkit routes transactions for different payment methods through various external payment providers. Any provider or issuing bank can fail at some point, and the team had no success-rate monitoring or alerting for individual payment instruments. Gateway dashboards show transaction states and overall success rate but offer no customizable alerts. Without alerts, customers keep paying with a failing instrument and have a bad experience.

Example from the article: the app shows customers a pop-up for payment methods with a low success rate, and a Slack alert fires when a method's 15-minute average success rate drops below its threshold.

Solution in brief

Every status update from a payment provider is written to InfluxDB as a point keyed by transaction ID and initiation time, so repeated updates overwrite the earlier status. Grafana computes success rate per payment method, gateway or card brand over a rolling window and sends Slack alerts when it drops below a per-method threshold.

Questions this case answers
  • How to detect failure of one of several external dependencies in near real time when each event reports its status several times and the vendor dashboards give no alerting
  • How to set alert thresholds on a ratio metric so that low traffic and brief dips do not cause false alarms
  • How to detect that one payment method or gateway has started failing before many customers hit it?
  • How to compute transaction success rate per payment method when each transaction sends several status updates?

Tools and methods

  • InfluxDB — time-series store that behaves as an upsert per transaction
  • Celery — asynchronous workers that write status updates to InfluxDB
  • Grafana — success-rate queries, dashboards and threshold alerts
  • Slack — alert delivery
  • Rolling-window threshold alert — 15-minute average success rate against a per-method threshold
Continue reading — it's free

Google or a one-time email code. No password.