Automating Receipt Collection: Apple Intelligence for On-Device Inference

Kabir Oberai13 Apr 2026

Problem

Ramp customers attach millions of receipts a month to card expenses in the iOS app, mostly by picking a photo by hand. Ramp wanted the app to scan the photo library and find receipts on its own. The photos are private, so all processing had to stay on the device. Plain string search over OCR text missed many valid receipts. Wrong suggestions hurt too: declined matches add noise and erode the user's trust.

Example from the article: the transaction says "Amazon Web Services" while the receipt prints "AWS", so an exact merchant-name search fails.

Solution in brief

Ramp uses a small on-device LLM (Apple FoundationModels) only for the fuzzy part, the merchant name. Date and amount are extracted by deterministic data detectors, because the small model invented numbers. An offline eval on a few hundred real transactions drove each iteration, and production accept and decline rates confirmed the result.

Questions this case answers
  • How to decide whether a noisy OCR text belongs to a given structured record when the data cannot leave the device and the model is small
  • How to split a matching task between deterministic extractors and a small LLM so that numeric fields are never hallucinated
  • How to match photos in a user's camera roll to card transactions without sending the photos off the device?
  • How to get usable precision and recall from a small on-device LLM that hallucinates numbers?

Tools and methods

  • Apple FoundationModels — on-device LLM with structured output (a Bool plus a free-form rationale)
  • Vision text recognition — OCR of each photo
  • String.dataDetectorMatches — deterministic extraction of dates and money amounts from OCR text
  • Hex — notebook that builds the eval CSV of transactions with receipt image identifiers
  • Structured answer schema — a rationale string generated before the verdict Bool
Continue reading — it's free

Google or a one-time email code. No password.