Problem
VK Ads needs categorical features for ad banners: readable labels from a nested tree of 790 categories (4 levels), several per banner, from text only. Annotators labeled 10 thousand banners, choosing up to three categories each. A TF-IDF + logistic regression model trained on them gave micro F1 0.60, and E5-large embeddings reached only 0.64, short of the 0.8 the team wanted. The cause was the labels: picking all relevant categories out of hundreds is hard for a person, so annotators missed topics.
Example from the article: for a banner about oral-care gadgets with a 10% promo code, annotators marked "Discounts and sales" and "Healthy lifestyle" but missed "Oral care" and "Electronics".
Solution in brief
Replace annotators with a self-hosted LLM and ask a different question: a yes/no for each {text, category} pair, walking down the tree level by level. Choose the model and prompting technique on a small hand-checked reference set. Few-shot prompts on Llama 3.1 Instruct won, and the same classifier trained on the LLM labels reached micro F1 0.80-0.81.
Questions this case answers
- How to label a large hierarchical category taxonomy for multi-label text classification when human annotators miss relevant labels
- How to replace human annotators with a self-hosted LLM that answers one yes-or-no question per category level
- How to build a training set for multilabel text classification with hundreds of categories when annotators miss labels
- How to label a large category taxonomy with a self-hosted LLM
Tools and methods
- Binary per-category labeling — one yes/no prompt per category.
- Few-shot prompting — two positive and two negative examples per prompt.
- Ollama — self-hosted runtime for the tested models.
- Pseudo-labeling — cleans the reference set.
- TF-IDF + logistic regression — light classifier to measure label quality.
Google or a one-time email code. No password.