Problem
Teams that label data by hand, often through crowdsourcing, get answers from people with their own experience and opinions. For objective tasks such as traffic signs, a reference exists and quality can be checked. For subjective tasks such as sentiment, translation or content filtering, ground truth is weak or absent. Labeling errors flow into the model and are multiplied there. A good instruction and selected annotators do not remove this, and without a reference set the team cannot see which answers are skewed.
Example from the article: In a retail pilot, annotators sorted fruit photos into "sale", "discount" and "write-off". For persimmon, 70% answered "sale" while the average predicted share was 40%: only some annotators know that dark stripes mean a good fruit.
Solution in brief
Bayesian Truth Serum (BTS) adds one field to the form: besides their own answer, each annotator estimates how others will answer. Comparing the actual distribution of answers with the predicted one exposes answers that are "surprisingly common" or "surprisingly uncommon". Gaps point to hidden knowledge, social pressure, sloppy work or an ambiguous question.
Questions this case answers
- How to find consensus among annotators on a subjective labeling task that has no reference answers
- How to detect ambiguous questions or unreliable annotators by asking each respondent to predict the distribution of others' answers
- How to find consensus among annotators when a labeling task has no ground truth
- How to detect hidden knowledge or bias in crowdsourced labels without a reference set
Tools and methods
- Bayesian Truth Serum — survey method that pairs an answer with a prediction of others' answers
- Surprisingly common / uncommon answers — actual share far from the predicted share
- Self-Assessment Manikin, PAD model — image scales used in the HSE case
- Toloka — crowdsourcing platform used for annotation
Google or a one-time email code. No password.
