From Chaos to Order: Extracting Structure from Unstructured Text with LLMs

Yulia Savateeva23 May 2026

Problem

Chestny Znak moderators check marketplace product cards against permit documents (certificates, declarations of conformity). One card as JSON can reach about 20 thousand tokens, so a human cannot compare it with the document fast enough for the stream. Errors cost both ways: a card passes without a valid permit, or an honest seller is rejected. The team needed an LLM check at up to 1 s per card and at least 80% accuracy.

Example from the article: the document covers fire-resistant protective equipment "except half-boots"; the card says "Toddler half-boots, color unicorn". The color can be ignored, but the product is for children and half-boots are excluded, so the card does not match.

Solution in brief

Split the check into narrow checks, each with one question and a small flat JSON schema. The LLM does only the semantic work (extract, normalize, judge); regular expressions, reference lists and Python set logic do the rest. Accuracy came from expert rule dictionaries and a field order with a short explanation before the verdict; a golden dataset measures it.

Questions this case answers
  • How to check whether a long structured record matches a permit document with an LLM at production latency by splitting one request into narrow single-question checks with deterministic logic around them
  • How to measure and raise the accuracy of LLM extraction against a golden dataset without fine-tuning
  • How to extract structured data from long documents with an LLM reliably enough for production
  • How to check a product card against a permit document with an LLM instead of manual moderation

Tools and methods

  • Qwen3-14B — chosen on the team's own benchmarks, run in no-think mode
  • Pydantic / JSON schema — output contract for each check
  • CatBoost / BERT classifier — gatekeeper: does the document fit the product group
  • Regular expressions and reference lists — run in parallel with the LLM checks
  • Golden dataset, MLflow — quality and consistency tracking
Continue reading — it's free

Google or a one-time email code. No password.