NLP at Alfa-Bank, Part 2: Data Labeling

Andrey (Data Scientist, Machine Learning Laboratory, Alfa-Bank)27 May 2023

Problem

A bank's NLP team needs training data, but in real business processes labeled data usually does not exist, or the annotation process is opaque and its quality doubtful. The model learns from the annotators, so it cannot beat the quality of the labels: if about 60% of the data is labeled accurately, a much higher model accuracy means the model has fitted the noise. Budget is fixed, and annotators want to finish fast, not accurately.

Example from the article: classifying chatbot messages by topic must stay in-house because messages may contain personal data, while classifying public social media reviews by sentiment can go to a crowdsourcing platform.

Solution in brief

Treat annotation as a managed process. Choose in-house or crowd labeling by data sensitivity and entry threshold, screen annotators with honeypots, and pick a representative sample. Label each example at least twice, measure quality against experts, and re-annotate as data drifts.

Questions this case answers
  • How to build a labeling process that gives reliable training labels when annotation quality caps model quality and the budget is fixed
  • How to choose a representative sample for annotation from a large unlabeled text corpus and keep labels accurate as data drifts
  • How to set up a data annotation process that gives a model reliable training labels
  • How to choose between in-house annotation and crowdsourcing

Tools and methods

  • Honeypot — expert-labeled control tasks mixed into the work to score and block annotators
  • Overlap annotation — at least two annotators per example, keep examples with agreement above a threshold
  • Hard sample selection, rare class mining, lexical mining — three ways to build a representative sample
  • Binary verification and blind re-annotation — two ways to estimate label quality
  • Toloka, Label Studio, ABC Elementary — crowdsourcing platform, open-source tool, boxed Docker solution
Continue reading — it's free

Google or a one-time email code. No password.