ML Module Architecture for Automating Technical Support for Pickup Point (PVZ) Staff

Dmitry Kondrashev, Yuriy Sofronov28 Oct 2023

Problem

Wildberries has a program for its order pickup points (PVZ), used by about 70 thousand operators and owners. Their questions went to two Telegram chats (6 thousand and 20 thousand members), where answers were hard to find and often never came, or to a ticket system where handling took from one day to several weeks. The team needed instant answers from regulations and news that change often.

Example from the article: the pilot's ChatGPT answered well when a person handed it the right document, but with automatic context selection and real questions, correct answers were very rare and the system was stopped after a couple of hours.

Solution in brief

Answer from a question-answer (QA) base first, using fuzzy and then vector search over stored questions. Only when nothing matches, compare the question with five stored keyword vectors per document chunk and let a self-hosted LLM answer from the best chunks. Both bases refresh automatically, and the LLM writes three question-answer pairs for every new source.

Questions this case answers
  • How to answer frequent user questions from a curated question base first and send only the remaining questions to a generative model over documents
  • How to retrieve context for a language model from short chunks using keyword vectors instead of whole-document vectors
  • How to build a product Q&A assistant over a FAQ base and internal documents with retrieval and an open-source LLM
  • How to answer typical questions instantly and send only the rest to an LLM

Tools and methods

  • PostgreSQL with pg_trgm — fuzzy matching of user questions to the QA base; stores both bases
  • multilingual-E5-large — embeddings of questions and keywords
  • SiberianFred-T5-instructor (FRED-T5) — self-hosted encoder-decoder LLM that answers from the context
  • TF-IDF + Linear SVC — classifier that tags each question with a business direction
  • Ray, Python parser, Telegram bot — LLM serving and source updates
Continue reading — it's free

Google or a one-time email code. No password.