Problem
Alfa-Bank collects ratings and comments from customers after service interactions. Ratings of 1-3 out of 5 count as negative, and experts call those customers and write a structured record of the complaint (an "expertise"). The customer experience team then has to assign each record to a negative category, and the categories feed several business lines. The list was built by hand from a small random sample, grew to 256 overlapping categories and confused operators, and some channels, such as debit cards, had no categories at all.
Example from the article: in 2022 the bank collected 9 million reviews from individuals, a little over a million of them negative, across 28 channels.
Solution in brief
Cluster all expertises of a channel to discover categories, organize them into a two-level hierarchy, and then train a BERT classifier on the resulting categories. Monthly three-person annotation of a random sample shows that the model agrees with the annotators at least as well as the operators do.
Questions this case answers
- How to build and maintain a hierarchical category taxonomy for free-text negative feedback by clustering all records instead of a small manual sample
- How to show that an automatic text classifier agrees with human annotators at least as well as the annotators agree with each other, measured on a recurring sample
- How to automatically categorize negative customer feedback at scale instead of manual expert sampling
- How to build and maintain a category taxonomy for customer complaints
Tools and methods
- Text clustering web service — proposes topics from all expertises of a channel (described in the author's earlier talk)
- Hierarchical categories — main negative (why the rating was low) and root negative (the pain left after the interaction)
- BERT classifier — assigns the categories automatically
- Training tricks — from the earlier talk
- Annotation with overlap of 3 — monthly quality check against humans
Google or a one-time email code. No password.