Classic Is All You Need! Selecting Reply Templates for Customer Requests

Dmitriy Bakanov25 May 2024

Problem

First-line support staff in an IT service management (ITSM) system must pick one of about 2,000 request templates for each ticket. The template sets the fields and the routing to a working group, so a wrong pick means a wrong group and a wrong answer to the client. The first line picks the right template in 60% of cases. The template set keeps changing, so a plain classifier over template IDs misses rare and new templates.

Example from the article: the service takes a ticket text and returns at most 10 templates sorted by relevance. Data: 5,740,031 tickets and 2,076 templates.

Solution in brief

Use classical NLP instead of BERT. A linear SVM on TF-IDF covers templates with enough history. Cosine similarity between the TF-IDF vectors of the ticket and of the template text covers rare and new templates. Both results are merged into one top-10 list, and a grid search keeps the model tuned.

Questions this case answers
  • How to recommend the top-k matching templates for a free-text request from a large, changing set of items when many items have few or no historical examples
  • How to choose between a transformer and a linear model with sparse features for a short-text ranking task under a CPU budget
  • How to recommend the top-10 matching ticket templates for a free-text support request
  • How to rank items when the item set changes and new items have no history

Tools and methods

  • TF-IDF with lemmatization — text features after cleaning and replacing personal data with tokens
  • Linear SVM — classifier on request-to-template history
  • Cosine similarity — match between ticket text and template text, no history needed
  • Grid search with block cross-validation — tuning on top-1, top-5 and top-10 accuracy
  • BentoML, FastAPI — packaging and serving the model on CPU
Continue reading — it's free

Google or a one-time email code. No password.