How we use LLMs with RAG in LemanaTECH's ecosystem of IT support ML models

Dmitry Terentev24 May 2025

Problem

LemanaTECH, a retailer with more than 40 thousand employees, runs IT support with more than 100 thousand tickets a month. Each ticket must go to the right one of 500+ business systems and 2,500+ service operations, and every next support line (L1 to L4) costs more per ticket. Routine questions that reach the expensive lines cost the most, and a ticket may wait up to 20 minutes untouched under the service rules.

Example from the article: employees ask routine questions about Kubeflow and Jenkins, and these tickets landed on the expensive data science and MLOps engineers of the L4 line.

Solution in brief

A chain of models handles each ticket. A BERT classifier with additive attention over ticket metadata routes it, a logistic regression decides whether a link to an instruction is enough, and for the MLOps class a RAG chatbot tries to answer first. If the bot fails, a ticket for an expert is created automatically.

Questions this case answers
  • How to route a high volume of short free-text requests into hundreds of classes using both the text and the metadata
  • How to decide which requests can be closed with a link to a reference and which need an expert, with a retrieval assistant answering first
  • How to route 100 thousand monthly IT support tickets into 500+ classes automatically
  • How to decide which tickets can be closed with a link to an instruction

Tools and methods

  • BERT with additive attention over features — ticket router, f1 = 0.89
  • TF-IDF + one-hot + boosting — baseline router, f1 = 0.86
  • Logistic regression — binary "can a link to an instruction close it" classifier
  • Qwen2.5 8B, quantized, on CPU — answer generation in RAG
  • multilingual-e5-large fine-tuned with triplet loss — retrieval over the knowledge base
Continue reading — it's free

Google or a one-time email code. No password.