NLP at Alfa-Bank, Part 7: Analyzing Customer Feedback

Eduard Yanakov27 May 2023

Problem

Alfa-Bank collects ratings and comments from customers after service interactions. Ratings of 1-3 out of 5 count as negative, and experts call those customers and write a structured record of the complaint (an "expertise"). The customer experience team then has to assign each record to a negative category, and the categories feed several business lines. The list was built by hand from a small random sample, grew to 256 overlapping categories and confused operators, and some channels, such as debit cards, had no categories at all.

Example from the article: in 2022 the bank collected 9 million reviews from individuals, a little over a million of them negative, across 28 channels.

Solution in brief

Cluster all expertises of a channel to discover categories, organize them into a two-level hierarchy, and then train a BERT classifier on the resulting categories. Monthly three-person annotation of a random sample shows that the model agrees with the annotators at least as well as the operators do.

Questions this case answers
  • How to build and maintain a hierarchical category taxonomy for free-text negative feedback by clustering all records instead of a small manual sample
  • How to show that an automatic text classifier agrees with human annotators at least as well as the annotators agree with each other, measured on a recurring sample
  • How to automatically categorize negative customer feedback at scale instead of manual expert sampling
  • How to build and maintain a category taxonomy for customer complaints

Tools and methods

  • Text clustering web service — proposes topics from all expertises of a channel (described in the author's earlier talk)
  • Hierarchical categories — main negative (why the rating was low) and root negative (the pain left after the interaction)
  • BERT classifier — assigns the categories automatically
  • Training tricks — from the earlier talk
  • Annotation with overlap of 3 — monthly quality check against humans
Continue reading — it's free

Google or a one-time email code. No password.