Problem
Store managers at Walmart receive hundreds of customer survey comments about cleanliness, staff, returns and other topics. Reading them by hand to see what changed over the last week, three months or six months takes too long. Managers also want to compare their store with others. A question-answering chatbot over the survey table ("survey playground") must search only the right stores and dates. Its answers must be checked for hallucination before managers trust them.
Example from the article: the table holds survey date, store number and customer comment; a manager asks for a feedback summary for one store over a chosen period.
Solution in brief
Embed every comment and store it in Milvus with the date and store number as scalar fields. At query time, filter by date range and store list first, run the vector search on the filtered rows, and pass the top results with the question to an LLM. Validate the pipeline with the RAG triad (context relevance, groundedness, answer relevance), scored by LLM graders in TruLens.
Questions this case answers
- How to answer free-text questions over a large table of short text records so that the search is limited by record metadata such as date and location
- How to measure whether a retrieval-augmented answer is supported by the retrieved context and addresses the question, without labeled answers
- How to let store managers ask questions about hundreds of customer survey comments and get a summary for a chosen store and period?
- How to restrict a RAG search to a date range and a list of stores before the semantic search?
Tools and methods
- Milvus — vector database that stores embeddings with scalar fields and supports filtered search
- Vertex AI text embeddings (Gecko) — 768-dimensional vectors for comments and queries
- GPT-4 via Walmart's Element platform — writes the final answer
- RAG triad — three LLM-graded checks: context relevance, groundedness, answer relevance
- TruLens with LiteLLM and Gemini — records test cases, runs graders, shows a dashboard
Google or a one-time email code. No password.