Problem
A finance engineering team at Spotify used LLMs to parse invoices from vendors around the world, in many languages and layouts. Rule-based parsers failed on the edge cases, but an LLM returns an answer without saying how sure it is. Finance controls (IT general controls and the Sarbanes–Oxley Act) require a score so that low-confidence fields go to a human and the rest pass automatically. The score had to cross specific thresholds before outputs could be trusted.
Example from the article: extracting invoice total (numeric) and invoice number (text) from invoices, where each field needs a confidence that tracks real accuracy.
Solution in brief
Ask five or six different LLMs the same question and take the answer most of them give. The confidence is the accuracy-weighted share of models that agree. Platt scaling then adjusts the raw score per field so it matches measured accuracy. Of three approaches tested, only voting correlated with accuracy.
Questions this case answers
- How to attach a calibrated confidence score to each field extracted by a generative model when the provider exposes no reliable uncertainty signal
- How to route model outputs to automatic acceptance or human review using a confidence threshold that regulators can audit
- How to get a trustworthy confidence score for each field an LLM extracts from a document?
- How to decide which LLM outputs can be auto-accepted and which go to a human reviewer?
Tools and methods
- Majority voting across several LLMs — picks the final value and gives the raw confidence score
- Accuracy-based weights — give better models more say and avoid ties
- Platt scaling — maps raw confidence to observed accuracy, per field
- Calibrator model and token logprobs — alternatives that were tested and rejected
Google or a one-time email code. No password.