Problem
A company that sends all LLM traffic through one gateway pays for several models and service tiers whose latency and failure rates keep changing. At Ramp, flex tier latency was not always higher than default, and one customer saw flex degrade between 9am and 5pm while matching standard at night. A static preference list cannot follow such patterns. For a flow with a loading spinner, a slow answer hurts as much as a failure.
Example from the article: an LLM reranker with a 6-second deadline had two fixed candidates (Gemini 3.1 Flash Lite Standard, GPT 5.4 Nano Standard); the list grew to five tier-specific candidates reordered per request.
Solution in brief
Track each candidate's failure rate with an exponentially weighted moving average (EWMA) and its log-latency with a Bayesian model sampled by Thompson sampling. For each request, compute the probability of a bad outcome (failure or missed deadline), combine it with relative cost into a score, sort the candidates and try them in order. Ramp reports 26.3% lower cost with no rise in errors.
Questions this case answers
- How to pick, per request, among interchangeable options with different cost and drifting latency and failure rates so that a deadline is met at the lowest cost
- How to choose per request between LLM providers and service tiers (default, flex, priority) when latency and failure rates change during the day?
- How to cut LLM spend without raising the error rate or missing a caller's deadline?
- How to reroute LLM traffic automatically when a provider starts rate limiting, and return when it recovers?
Tools and methods
- Thompson sampling — draws a plausible latency distribution per candidate, so routing keeps exploring.
- Normal-Inverse-Gamma (NIG) posterior — conjugate model of the mean and variance of log-latency.
- EWMA of failure rate — tracks provider-side failures such as rate limiting.
- Redis — holds the running statistics shared across gateway instances.
Google or a one-time email code. No password.