Problem
Blinkit routes transactions for different payment methods through various external payment providers. Any provider or issuing bank can fail at some point, and the team had no success-rate monitoring or alerting for individual payment instruments. Gateway dashboards show transaction states and overall success rate but offer no customizable alerts. Without alerts, customers keep paying with a failing instrument and have a bad experience.
Example from the article: the app shows customers a pop-up for payment methods with a low success rate, and a Slack alert fires when a method's 15-minute average success rate drops below its threshold.
Solution in brief
Every status update from a payment provider is written to InfluxDB as a point keyed by transaction ID and initiation time, so repeated updates overwrite the earlier status. Grafana computes success rate per payment method, gateway or card brand over a rolling window and sends Slack alerts when it drops below a per-method threshold.
Questions this case answers
- How to detect failure of one of several external dependencies in near real time when each event reports its status several times and the vendor dashboards give no alerting
- How to set alert thresholds on a ratio metric so that low traffic and brief dips do not cause false alarms
- How to detect that one payment method or gateway has started failing before many customers hit it?
- How to compute transaction success rate per payment method when each transaction sends several status updates?
Tools and methods
- InfluxDB — time-series store that behaves as an upsert per transaction
- Celery — asynchronous workers that write status updates to InfluxDB
- Grafana — success-rate queries, dashboards and threshold alerts
- Slack — alert delivery
- Rolling-window threshold alert — 15-minute average success rate against a per-method threshold
Google or a one-time email code. No password.
