Problem
A team needs labels for a fine-grained taxonomy, here 32 speech-function classes for dialogue systems, but linguists are too expensive to label a whole corpus. Crowdworkers lack the linguistic background to pick one class out of 32. Many also click through answers to get paid faster, which adds noise. Nobody knows whether crowdworkers or an LLM can replace the experts.
Example from the article: annotating dialogues from DailyDialog with labels such as React.Rejoinder.Support.Track.Clarify. Three linguists build a 64-dialogue gold standard, crowdworkers and ChatGPT then label the same dialogues.
Solution in brief
Experts turn the taxonomy into a tree of 2–4-option questions, so a non-expert reaches the class without knowing its name. Crowdworkers walk the tree through a custom interface with automatic honesty checks. ChatGPT walks the same tree node by node as a virtual annotator. All three are compared to the expert gold standard by Fleiss' kappa and recall, precision and F1.
Questions this case answers
- How to get reliable labels from non-expert annotators for a fine-grained taxonomy by turning flat classes into a tree of simple questions, and compare humans with an LLM annotator against an expert gold standard
- How to filter out careless annotators automatically without reference labels
- How to annotate a complex 32-class taxonomy when annotators are not domain experts
- How to get usable labels from crowdworkers and filter out dishonest ones
Tools and methods
- Tree-based annotation scheme — a chain of simple questions that ends in one class.
- Toloka — crowdsourcing platform, used through the API with a custom interface.
- Basic validation and gold-set validation — rule flags and agreement with reference labels.
- ChatGPT as a tree-walking annotator — one prompt per node of the tree.
- Fleiss' kappa, weighted recall and precision, macro F1 — quality and agreement metrics.
Google or a one-time email code. No password.