Streaming Incident Clustering for Faster Handling by Operators

Ivan Oleynikov25 May 2024

Problem

An enterprise IT service desk receives up to 25 incidents per second, about 160,000 a month. Operators open every incident separately, so many incidents with one cause are handled one by one. Only 10% of incidents were linked to problems, against a target of 85%. The service had to find similar incidents in the stream and form groups that an operator handles as one object. Constraints: online processing, CPU-only inference and retraining, and new groups that keep appearing.

Example from the article: an incident from email with a template header (company, tax ID, VIP flag) and a free-form description of a bank statement export in the "1C" format that pulls in wrong operations.

Solution in brief

Split the stream by business attributes (workgroup and service) into metaclusters. Inside each one, vectorize texts with TF-IDF reduced by SVD to 300 dimensions and run a modified Leader clustering: an incident joins the closest cluster if cosine similarity exceeds that cluster's threshold R, otherwise it starts a new cluster. TF-IDF beat Navec and BERT.

Questions this case answers
  • How to group a stream of short texts into clusters online with a fixed similarity radius per cluster, when there is no fixed set of classes and only CPU is available
  • How to choose a text representation for online clustering when the encoder cannot be retrained
  • How to group similar incoming tickets or incidents into clusters online, on CPU only
  • How to cluster a text stream when new groups appear all the time and there is no fixed set of classes

Tools and methods

  • Leader clustering — online fixed-radius clustering, adapted to run inside metaclusters
  • TF-IDF + truncated SVD (300 dimensions) — CPU-friendly text vectors, no neural retraining
  • Cosine similarity — closeness of an incident to a cluster center
  • Metacluster (workgroup + service) — first-level split by business attributes
  • BIRCH, Quality Threshold — rejected candidates
  • Navec, BERT — embedding baselines
Continue reading — it's free

Google or a one-time email code. No password.