Introducing a RAG-based bot to streamline CS

Chaeseung Lee4 Jun 2025

Problem

The owners of an internal automation platform (AWX at LY Corporation, the operator of the LINE app) get the same questions again and again. Users start work without reading the guides, and a long guide set makes the right page hard to find. Every repeated question costs the administrators operating time. Users write in several languages, since the teams sit in different countries.

Example from the article: in the bot demo (Fig. 1) a new user reports infinite redirects when opening AWX pages and asks the administrator.

Solution in brief

A Slack bot answers each inquiry first. It embeds the question, finds the nearest chunks in two vector indexes (wiki guides and past resolved Slack threads), and gives them to an LLM as context. The user then presses "Issue Resolved" or "Call AWX Admin", so a human steps in only when the bot fails.

Questions this case answers
  • How to deflect repeated support questions from a small owner team using existing guides and past resolved threads, when the questions arrive in several languages
  • How to cut repeated questions to the owners of an internal platform with a first-line RAG bot?
  • How to choose which past support threads and wiki pages go into the retrieval index?
  • How to let users escalate to a human when the bot answer does not help?

Tools and methods

  • Retrieval-augmented generation (RAG) — supplies the LLM with trusted context along with the question.
  • paraphrase-multilingual-mpnet-base-v2 (Sentence Transformers) — multilingual embeddings for questions and documents.
  • OpenSearch — vector store with k-NN search, two indexes.
  • LangChain — LLM orchestration.
  • OpenAI API — answer generation.
  • Slack Bolt for Python — bot framework, with a Slack workflow form for inquiries.
Continue reading — it's free

Google or a one-time email code. No password.