Problem
Airbnb merged separate support-issue taxonomies, used by guests and Hosts, agents and ML models, into one called Contact Reasons. A new taxonomy is often built before real data or a downstream workflow exists. Production feedback arrives only after rollout, model retraining and logging, which takes several quarters. Before this work the team judged taxonomies by expert review and user research, which are subjective and small-sample, and small edits such as adding a node were never tested.
Example from the article: the old taxonomy had only "Host-initiated" and "Guest-initiated" cancellation nodes, so it could not tell when the person asking for help was not the one who cancelled.
Solution in brief
T-LEAF scores a taxonomy offline on three numbers: coverage (how few items fall into "other"), usefulness (whether the number of categories fits the data size) and agreement (whether different labelers pick the same category). Agreement is measured either with several human annotators or, much cheaper, as the training accuracy of a model fitted on single-rater labels. The scores are computed on each taxonomy version before launch.
Questions this case answers
- How to score the quality of a label hierarchy before deployment, when downstream feedback takes several quarters to arrive
- How to check that a category scheme is neither too coarse nor too granular for the data it must describe
- How to measure the quality of a new classification taxonomy before it is launched in production?
- How to tell whether a taxonomy has too few or too many categories?
Tools and methods
- Coverage score — share of items not labeled "other" or "undefined".
- Usefulness score — compares the number of categories with the target for benchmark items.
- Cohen's kappa — agreement between human annotators who label the same items.
- Model training accuracy — cheaper proxy for agreement, from a classifier trained on single-rater labels.
Google or a one-time email code. No password.