Problem
Finance teams approve a high volume of expenses by hand, and each one needs a judgment against a written policy. An LLM agent can take over most of it, but one low-quality or unexplained decision costs the product its users' trust. Ramp built a "policy agent" for expense approval and had to make it explainable, aware of its limits and under the customer's control.
Example from the article: a receipt for golfing was submitted under the wrong transaction, and the built-in escape hatch let the agent explain that mismatch.
Solution in brief
The agent shows a short reason for every decision with links to the policy text it relied on. It returns one of three categories instead of a confidence score, and "needs review" falls back to the old human process. Customers set the autonomy level in the workflow builder, add deterministic rules on top, and edit the policy when they disagree. Evals use golden datasets reviewed by the team.
Questions this case answers
- How to make an LLM agent that takes decisions on behalf of users trustworthy and controllable when wrong outputs are costly
- How to evaluate a decision agent when the user feedback used as ground truth is biased
- How to make users trust an LLM agent that approves or rejects expenses on their behalf?
- How to let an LLM decision agent say it is unsure without hallucinating an answer?
Tools and methods
- Human in the loop — needs-review cases go to the usual escalation; agents start as suggestions
- Rule-based guardrails — dollar limits, vendor blocklists and category restrictions layered over the LLM decision
- Golden set — datasets reviewed by the team, used instead of raw user approvals as ground truth
Google or a one-time email code. No password.