Problem
CLA, an accounting and professional services firm, has auditors who read SOC (system and organization control) reports from clients' vendors. A report runs 50 to 100+ pages, with unstructured sections and a format that varies by issuing audit firm. Manual extraction took up to 8-10 hours per report and left missed fields, typos and person-to-person variation. An LLM workflow now takes under 10 minutes, but at volume reviewers cannot check every field with equal care, and nobody can prove six months later that quality has held.
Example from the article: the speakers ask what happens if the system extracts a thousand SOC documents a day: auditors would face 10,000+ fields to verify.
Solution in brief
An LLM judge gives each extracted field a confidence score. It needs no ground truth, guides reviewers and flags drift for the AI team. Reviewer corrections become ground truth, and custom MLflow scorers compare them with model output section by section to get precision, recall and F1 per run.
Questions this case answers
- How to measure the quality of LLM extraction of structured fields from long documents when no ground truth exists at the start
- How to score nested structured output section by section when list items have no stable ID and equivalent strings differ in punctuation
- How to measure the quality of LLM extraction of structured data from long documents
- How to tell reviewers which extracted fields need attention without ground truth
Tools and methods
- LLM judge confidence score — per-field triage signal and drift monitor
- Feedback-driven evaluation — corrections become evaluation data
- Custom scorers (MLflow) — exact, fuzzy or LLM comparison per field
- Object matching for lists — align items before scoring fields
- Databricks Apps, Lakebase, Lakeflow, MLflow — UI, storage, pipeline, tracking
Google or a one-time email code. No password.