LLMs vs. Annotators: How to Get High-Quality Text Labeling

Elena Lebedeva1 Nov 2024

Problem

VK Ads needs categorical features for ad banners: readable labels from a nested tree of 790 categories (4 levels), several per banner, from text only. Annotators labeled 10 thousand banners, choosing up to three categories each. A TF-IDF + logistic regression model trained on them gave micro F1 0.60, and E5-large embeddings reached only 0.64, short of the 0.8 the team wanted. The cause was the labels: picking all relevant categories out of hundreds is hard for a person, so annotators missed topics.

Example from the article: for a banner about oral-care gadgets with a 10% promo code, annotators marked "Discounts and sales" and "Healthy lifestyle" but missed "Oral care" and "Electronics".

Solution in brief

Replace annotators with a self-hosted LLM and ask a different question: a yes/no for each {text, category} pair, walking down the tree level by level. Choose the model and prompting technique on a small hand-checked reference set. Few-shot prompts on Llama 3.1 Instruct won, and the same classifier trained on the LLM labels reached micro F1 0.80-0.81.

Questions this case answers
  • How to label a large hierarchical category taxonomy for multi-label text classification when human annotators miss relevant labels
  • How to replace human annotators with a self-hosted LLM that answers one yes-or-no question per category level
  • How to build a training set for multilabel text classification with hundreds of categories when annotators miss labels
  • How to label a large category taxonomy with a self-hosted LLM

Tools and methods

  • Binary per-category labeling — one yes/no prompt per category.
  • Few-shot prompting — two positive and two negative examples per prompt.
  • Ollama — self-hosted runtime for the tested models.
  • Pseudo-labeling — cleans the reference set.
  • TF-IDF + logistic regression — light classifier to measure label quality.
Continue reading — it's free

Google or a one-time email code. No password.