History of the Unified Classifier Development

Andrey27 May 2023

Problem

A bank's chatbot hands hard cases to a human operator. After the call the operator fills in a classifier: product, topic, subtopic, resolution status and, if needed, the channel. Together product, topic and subtopic give more than 1000 classes. Analysts build topic-share reports on this data, so the labels matter. Filling them in is routine work that adds to average handling time (AHT), and label quality depends on the operator's experience.

Example from the article: a first baseline trained on messages within ±1 hour of the moment the operator saved the topic reached only about 40% accuracy.

Solution in brief

Audit the operator labels first, then relabel whole dialogues in a controlled process. Fine-tune one BERT-type model with six classification heads and mask impossible class combinations at inference. Score yesterday's dialogues in a daily batch job. For the weakest target, product, mask products the customer does not hold.

Questions this case answers
  • How to classify free-text messages into a very large hierarchical label set when the training labels were attached to the wrong text and must be audited first
  • How to stop a multi-head classifier from predicting combinations that are invalid under a known hierarchy or under each entity's eligibility list
  • How to automate ticket classification into 1000+ hierarchical classes in a contact center
  • How to audit the quality of operator-made labels before training a classifier

Tools and methods

  • RuBERT — encoder, fine-tuned on bank data, with six fully connected heads trained jointly (multitask)
  • Label audit — binary recheck and blind relabeling with agreement measurement
  • Label Studio — labeling interface for cross-annotation
  • Logit masking — minus infinity for impossible class combinations
  • Airflow and MLflow — daily batch scoring with the latest registered model
Continue reading — it's free

Google or a one-time email code. No password.