LLMs and Other NLP Models in the Service of CRM and Customer Journey Improvement

Anna Shirshova24 May 2025

Problem

A bank's CRM team had to extract what customers say about savings products from two weeks of call-center transcripts and chatbot data. The business gave no criteria for the result, the customer pool exceeds 100 million, and production limits require interpretable, not-too-complex models. Raw transcripts are full of typos and operator scripts. Experts find the true cause of each contact by hand, so such contacts are processed with a two-week delay.

Example from the article: about 150 thousand messages per product over two weeks, about 50 words each. Collocations with weak preprocessing returned "can't hear well" instead of anything the business could act on.

Solution in brief

Embed cleaned customer texts with BGE-M3, cluster them (UMAP, then KMeans), and describe each cluster by its top collocations. Then give an LLM (Qwen2.5-7B-Instruct) the texts nearest the cluster centroid that contain those collocations and ask for a one-sentence summary. The same embeddings feed a root-cause classifier and a sentiment model.

Questions this case answers
  • How to find what customers complain about in large unlabeled text collections and summarize clusters that do not fit into a language model context
  • How to classify contacts by root cause rather than by topic, using only the text, when experts label causes by hand
  • How to find what customers complain about in call-center and chatbot transcripts without labeled data
  • How to summarize the topics of a large pile of customer contacts when it does not fit into an LLM context

Tools and methods

  • BGE-M3 — multilingual text embeddings, the shared base for all three tasks
  • UMAP + KMeans — dimensionality reduction and clustering of embeddings
  • PMI collocations — 3- and 4-word phrases that label each cluster
  • Qwen2.5-7B-Instruct — one-sentence summary of each cluster
  • PCA + logistic regression — root-cause classifier over 86 classes
  • PCA + CatBoost — two-class sentiment model
Continue reading — it's free

Google or a one-time email code. No password.