Problem
A team that builds an ML classifier needs ground-truth labels that match its own business definitions. Open datasets and logged data rarely fit, and old labeling guidelines are often too broad or outdated. Projects stall on the lack of reliable labeled data, and the process of collecting it is rarely documented. The common reflex is to outsource annotation from day one, which trades off quality and fit with your guidelines against speed and cost.
Example from the article: trivago needed a training set for a model that assigns accommodation types (Hotel, Guest House, Serviced Apartment and so on) to newly imported properties. Advertisers' own type mappings were missing or interpreted differently, and the old guidelines no longer covered new accommodation kinds such as apartments.
Solution in brief
Treat annotation as an iterative process with six checkpoints: guidelines, inter-annotator agreement (IAA), calibration sessions, platform and task design, automation, and a trained annotator group with a feedback channel.
Questions this case answers
- How to build a labeled dataset with consistent labels when category definitions are subjective, differ between regions and drift as the catalog changes
- How to decide between outsourcing annotation and a small trained annotator pool when data has many edge cases
- How to get reliable ground-truth labels for a new classifier when the old labeling guidelines are vague?
- How to measure and raise agreement between annotators before scaling a labeling project?
Tools and methods
- Annotation guidelines — written rules with examples and edge cases, revised after every round.
- Inter-annotator agreement (IAA) — an agreement metric computed on a shared sample labeled by a fixed pool of annotators.
- Calibration sessions — meetings where annotators explain their decisions and the guidelines are updated.
- Annotation platform with an API — scripted task creation and result export.
Google or a one-time email code. No password.