Financial Benchmarks

Kedar Thakkar, Anton Biryukov, Ashwin Kumar, Ryne Carbone9 Mar 2026

Problem

Ramp ships LLM features for finance teams: reading invoices, extracting numbers from financial statements, coding expenses, checking policy and compliance rules, and routing transactions to funds. New models arrive every few months, and the team must know which one fits each task before changing production. A wrong pick costs accuracy, money or latency, and several tasks have no clean ground truth.

Example from the article: an invoice counts as correct only if the customer makes zero edits, and a business that always builds the invoice number as "INV" plus the date (INV-01012026) expects that pattern even when the number is absent from the document.

Solution in brief

For each product task, build a benchmark from real production data with a metric that matches how the customer judges the result. Run every model at several reasoning levels, plot quality against cost or latency, and choose from the Pareto frontier. Tasks without ground truth get two opposing metrics, and tasks with asymmetric errors get a blended score.

Questions this case answers
  • How to select a model per task from a pool of candidates when accuracy, unit cost and latency pull in different directions and ground truth is partial or contested
  • How to evaluate a model that may abstain when the only reference is past human decisions and the two error types have asymmetric costs
  • How to choose an LLM for each product task when no single model wins on accuracy, cost and latency?
  • How to evaluate an LLM that decides approve, reject or unsure when there is no clean ground truth?

Tools and methods

  • Task-specific benchmarks on real data — one metric per task, taken from the customer's final decision
  • Pareto frontier — cost or latency against quality; keeps models no other model beats on both
  • Tolerance thresholds — match rate at 1%, 5% and 10% relative error
  • Opposing metrics and composite score — disagreement vs. unsure rate; Compliance Adherence Score
  • Hand-curated hard set — non-trivial routing cases with reliable ground truth
Continue reading — it's free

Google or a one-time email code. No password.