Problem
Chestny Znak moderators check marketplace product cards against permit documents (certificates, declarations of conformity). One card as JSON can reach about 20 thousand tokens, so a human cannot compare it with the document fast enough for the stream. Errors cost both ways: a card passes without a valid permit, or an honest seller is rejected. The team needed an LLM check at up to 1 s per card and at least 80% accuracy.
Example from the article: the document covers fire-resistant protective equipment "except half-boots"; the card says "Toddler half-boots, color unicorn". The color can be ignored, but the product is for children and half-boots are excluded, so the card does not match.
Solution in brief
Split the check into narrow checks, each with one question and a small flat JSON schema. The LLM does only the semantic work (extract, normalize, judge); regular expressions, reference lists and Python set logic do the rest. Accuracy came from expert rule dictionaries and a field order with a short explanation before the verdict; a golden dataset measures it.
Questions this case answers
- How to check whether a long structured record matches a permit document with an LLM at production latency by splitting one request into narrow single-question checks with deterministic logic around them
- How to measure and raise the accuracy of LLM extraction against a golden dataset without fine-tuning
- How to extract structured data from long documents with an LLM reliably enough for production
- How to check a product card against a permit document with an LLM instead of manual moderation
Tools and methods
- Qwen3-14B — chosen on the team's own benchmarks, run in no-think mode
- Pydantic / JSON schema — output contract for each check
- CatBoost / BERT classifier — gatekeeper: does the document fit the product group
- Regular expressions and reference lists — run in parallel with the LLM checks
- Golden dataset, MLflow — quality and consistency tracking
Google or a one-time email code. No password.
