Problem
An oil and gas client gets many letters in a shared mailbox, and one specialist routes each to one of 36 departments by hand. The client wants to cut that workload, not replace the person. The data is hard: about 1,800 letters, mostly scanned PDFs without a text layer, formal language, industry jargon and a long tail of rare classes. Trained on the raw texts, a model handles only roughly the first 10 classes well.
Example from the article: in a blind test on 36 new letters the model made 8 errors, and only 2 of them were genuine model mistakes.
Solution in brief
Read the scans with a vision-language model instead of classic OCR, then grow the rare classes with LLM generation, paraphrasing and back-translation until synthetic texts are up to about 50% of the data. Fine-tune a Russian BERT (ruRoberta-large worked best) with focal loss, gradual unfreezing and a larger learning rate for the head. The service returns the top 3 departments, and the specialist makes the final call.
Questions this case answers
- How to classify scanned documents into dozens of routing classes when the rare classes have very few examples
- How to keep a text classifier from relying on entity names that break when people change roles
- How to classify scanned business letters into dozens of departments with only about 1,800 labeled examples
- How to handle a long tail of rare classes in text classification with LLM-based augmentation
Tools and methods
- Qwen2.5-VL-7B — OCR of scanned PDFs, followed by regex cleanup of artifacts
- Qwen2.5-32B — generation, paraphrasing and polishing of back-translated texts for rare classes
- Helsinki-NLP — back-translation
- rubert-tiny2, ruRoberta-large, RuBERT, RuModernBert — classifiers
- Focal loss, stratified batches, differential learning rate, gradual unfreezing — training for imbalanced classes
- Macro F1, balanced accuracy — metrics
Google or a one-time email code. No password.