Majority vote for CV annotations: improve your data labeling pipeline

Karina Kvantchiani27 May 2023

Problem

A team building computer vision datasets on crowdsourcing platforms gets labels from annotators who are weakly motivated by the result. To compensate, every sample is labeled by several annotators (overlap) and the answers must be merged into one final annotation. For classification this is a simple vote. For boxes, time intervals and masks it is hard: annotators draw boxes in the wrong place, set the wrong label, skip objects or try to finish the task fast. The usual IoU check discards or re-sends many sets, which costs time and money, and still lets bad boxes through.

Example from the article: in the HaGRID gesture dataset three annotators drew a box correctly and a fourth drew a small one. The IoU check passes, but the averaged box shifts slightly downward.

Solution in brief

Aggregate in two stages. Hard-aggregation strictly checks that annotations agree on box count, cluster, label and IoU, then averages. Annotations that fail go through soft-aggregation, which removes junk, fixes minority labels and fills missed boxes using a 70% majority threshold, then retries the hard check.

Questions this case answers
  • How to merge overlapping crowdsourced annotations of boxes, intervals and masks into one reliable label when annotators are unmotivated and sometimes careless
  • How to aggregate annotations in two stages, checking structural agreement before overlap, so that one bad annotation does not discard a whole set
  • How to merge overlapping crowdsourced bounding-box annotations into one reliable box
  • How to aggregate overlapping time-interval and segmentation-mask annotations from several annotators

Tools and methods

  • Overlap annotation and majority vote — several annotators per sample, merged by agreement
  • IoU threshold — similarity check between annotations
  • Clustering — 2D for boxes, 1D for time intervals, to split annotations into per-object groups
  • Task decomposition — one segmentation class per task
  • Yandex Toloka, ABC Elementary — crowdsourcing platforms used for the first three pipeline stages
Continue reading — it's free

Google or a one-time email code. No password.