Method

Inter-annotator agreement

Kappa and IoU agreement between annotators.

11 cases from 11 companies apply Inter-annotator agreement. 7 report results in production.

Typical tasks: Evaluate models, Label data with LLMs, Predict long-term effect from short tests. Companies: Netflix, Atlassian, Alfa-Bank, trivago, Airbnb, Celsus and 5 more. Often combined with: Golden set / benchmark, LLM-as-a-judge, Overlap annotation, Human / expert labeling, Prompt optimization and 16 more.