Analytics, Data, Metrics and Motivation: How to Build a Data Labeling Process in a Large Bank

Roman Kuznetsov24 May 2025

Problem

A bank's labeling team produces training data for many internal customers (call center dialogues, support chat messages), each with its own instructions. A very large number of labelers do monotonous work, so speed and quality drift, and managers have no objective way to set volume targets or to see which tasks are slow. Targets set "by eye" waste capacity. Instructions change too often for classic quality checks, and labelers are expected to deliver more for the same pay.

Example from the article: on an old text-labeling project the target had been set by eye; recalculating it properly showed that the efficiency of the labeling resource could grow by 20%.

Solution in brief

Analysts set targets from task times with abnormally long cases cut off, flag complex tasks from time distributions, and score quality by a gold standard and agreement. Data scientists route simple tasks to an LLM plus one human. A weighted rating and achievements motivate labelers without extra pay.

Questions this case answers
  • How to set fair daily productivity targets for a human labeling workforce from task-time distributions when long tails come from distractions
  • How to measure and monitor annotation quality when instructions change often, and route simple cases to a model with one human check
  • How to set fair daily productivity targets for a data-labeling team
  • How to measure and monitor labeling quality when instructions change often

Tools and methods

  • Trimmed average task time — a per-task time threshold removes distracted cases before computing the daily target
  • Gold-standard comparison — group leads label a sample; other labelers are scored against it
  • Agreement — share of tasks where labelers' labels match; only agreed cases go to training
  • LLM task routing — a model flags simple tasks for "LLM + human" labeling
  • Weighted labeler rating — volume, quality, manager's assessment, initiative
Continue reading — it's free

Google or a one-time email code. No password.