Application of NLP for Customer Satisfaction Assessment and Optimization of SMS Traffic

Anna Shirshova25 May 2024

Problem

A bank's modeling team gets text tasks from the business. Survey analytics is due within a day. What do customers write in satisfaction surveys? What do they say online about the bank and its competitors? Which SMS messages could be sent cheaper? Without text analysis, comments stay unread and the processes that anger customers stay unnoticed.

Example from the article: many departments send SMS, and nobody knew which messages follow a template. Operators charge less for templated messages, so template search over a pilot sample of 1.5 billion records saved several million rubles per month.

Solution in brief

Start with cheap frequency methods. Rank collocations (stable word combinations) per rating class or per review scenario, and put most of the effort into text cleaning and exclusion lists. For SMS, replace variable parts with tags and group messages by their sequence of POS and NER tags. LLM summarization was only being tested and depended strongly on the prompt.

Questions this case answers
  • How to find what customers complain about in free-text feedback within a day, using frequent word combinations per segment with explainable statistics
  • How to group short automated messages into templates by their structure rather than their meaning, to find the cheapest sending option
  • How to find what customers complain about in free-text survey answers within a day
  • How to monitor online reviews of a bank and its competitors by product and sentiment at scale

Tools and methods

  • Collocations by log-likelihood ratio (LLR) — top phrases per rating class and per review scenario.
  • LDA, TF-IDF logistic regression — topics and word relevance per rating in survey answers.
  • Word2Vec weighted by TF-IDF, OPTICS — clusters of free-text answers.
  • n-grams, FPGrowth — frequent word sequences in SMS.
  • slovnet POS and NER tags — structural templates of SMS.
  • Saiga, t5, rugpt3, rut5 summarizers — trial summaries of reviews.
  • Spark — all large-volume pipelines.
Continue reading — it's free

Google or a one-time email code. No password.