Problem
A telecom chatbot classifies subscriber messages into known intents such as «find out expenses» or «change tariff». Everything that fits no class lands in «other», and new user needs hide there. Analysts need to find them by clustering that class, but a clustering pipeline has many free choices (embedding model, pooling, algorithm, hyperparameters) and no labels to score them. Classical internal metrics also mishandle clusters of arbitrary shape and ignore noise points.
Example from the article: MegaFon clusters the «other» class, creates new intent classes from the clusters, and retrains the classifier on old and new classes.
Solution in brief
Embed messages with Sentence-Transformers, reduce dimensionality with UMAP, and cluster with HDBSCAN. Pick the pipeline without labels: tune hyperparameters on an internal metric (silhouette or DBCV made robust to noise), then translate the metric into an expected external score (AMI) with regressions fitted on labeled data.
Questions this case answers
- How to discover new categories in a stream of unlabeled short texts left over by an existing classifier
- How to select a clustering pipeline and its hyperparameters without labels when the clusters have arbitrary shape and some points are noise
- How to find new user intents in chatbot messages that the intent classifier assigned to the «other» class?
- How to choose an embedding model and a clustering algorithm for short texts when there are no cluster labels?
Tools and methods
- Sentence-Transformers — sentence embeddings (transformer plus mean or CLS pooling); four models compared.
- UMAP — dimensionality reduction before clustering.
- HDBSCAN — clustering with automatic cluster count, noise and varying densities.
- BIRCH — alternative clustering, compared against HDBSCAN.
- Isolation Forest — outlier detection, compared against HDBSCAN noise.
- DBCV and silhouette — internal metrics, modified to account for noise.
- AMI (adjusted mutual information) — external metric used as ground truth.
Google or a one-time email code. No password.
