Problem
Chestny Znak, the Russian product-labeling system, digitizes more than 1,000 pages of documents per day for search, LLM analysis and routing. The documents are confidential, so OCR runs on the company's own hardware. Russian adds Cyrillic and Latin look-alike glyphs (s/c, o/o, r/p), and the pages are full of tables with merged cells and multi-level headers. Every open-source engine claims the best quality, and a wrong pick means bad tables or a GPU bill tens of times higher than CPU.
Example from the article: the team compared 10 open-source solutions on 91 real company documents (4,843 pages) and picked DeepSeek-OCR-2 for GPU and its own converter for CPU-only, table-heavy cases.
Solution in brief
Get ground truth for free: export text from .docx files, convert them to PDF and strip the text layer. Score text, table detection, table structure and cell text separately, and measure speed and throughput on one fixed server. Merge the results into a summary table and an average rank.
Questions this case answers
- How to build a ground-truth benchmark for document text extraction without manual labeling by exporting text from source files
- How to score text, table detection and cell content separately when comparing extraction engines on complex tables
- How to choose an OCR engine for Russian documents with complex tables when documents cannot leave the company
- How to build an OCR benchmark with ground truth without manual labeling
Tools and methods
- Ground truth from DOCX — export text, render PDF, remove the text layer, run OCR, compare.
- CER and CER AlphaNum-only — character error rate with and without formatting.
- Table metrics — detection, structure (rows × columns), cell text CER.
- CPU pipelines — DocsConvertor, Docling, marker.
- GPU vision-language models — DeepSeek-OCR(-2), chandra(-ocr-2), dots.mocr, Qwen-VL.
Google or a one-time email code. No password.
