Problem
A bank supports small and medium businesses through four channels: chats, calls (transcribed by ASR), emails and forms. Hundreds of thousands of requests arrive per month. Operators assign one topic, and the labels are noisy because the taxonomy is large. Analysts then check by hand which topics grew and whether technical problems occurred. The customer wanted more attributes per request, detection of new topics, and one search window.
Example from the article: sentiment with five classes (insults, threats/risks, negative, positive, neutral) had 30 negative requests and 30,000 unlabeled ones.
Solution in brief
Small models do most of the work. An LLM handles what they cannot: labeling, uncertain cases, new topics and search checks. Labels are bootstrapped from regex and LLM pseudo-labeling. A light CatBoost model hands uncertain requests to a heavy LLM. Topics come from three sequential BERT-based models, one per topic level, with channel-specific logit masking.
Questions this case answers
- How to route and label short customer requests across several channels with a cheap model and an LLM only for uncertain cases
- How to assign several topics per request across taxonomies that differ by channel when historical labels hold only one topic
- How to build natural-language search over labeled requests with filters, hybrid retrieval and LLM verification
- How to build a sentiment classifier for support requests when there are only 30 labeled examples
Tools and methods
- Regex + LLM few-shot/CoT labeling, pseudo-labeling — build a training set from 30 examples
- CatBoost + SBERT embeddings + text features → LLM cascade — sentiment with per-class confidence thresholds
- rubert-tiny-turbo, three heads, logit masking, softmax with a threshold — multi-level, multi-topic classification
- LLM with deduplication and popularity check — new-topic discovery (BERTopic was tried first)
- LLM-as-a-Judge + BERTScore — benchmark for summarization prompts
- Text2Filters, BM25 + embeddings, reranker, LLM verification — natural-language search
Google or a one-time email code. No password.