Problem
Search at delivery and retail apps relies on a knowledge graph (KG) of dishes, products and merchants for query expansion and ranking. LLMs and scrapers add edges from text co-occurrence, so wrong parent-child links appear. Annotators cannot keep up with millions of entities that change daily. A wrong edge makes query expansion return irrelevant items, which lowers click-through rate (CTR) and user trust.
Example from the article: an LLM links "Pho" to "Italian Noodle Soup" because the words look alike, and a search for that parent then shows irrelevant dishes.
Solution in brief
Treat every unverified edge as a hypothesis. Show it to real users in a low-risk search slot next to verified results, log the impression and the user's reaction, and score each edge by weighted engagement per impression. Edges above a threshold become verified, edges below a second threshold are deleted.
Questions this case answers
- How to validate machine-generated relations between catalog entities continuously, when manual review does not scale and offline ground truth does not exist
- How to collect user feedback on unproven recommendations at a bounded cost to the user experience
- How to check whether LLM-generated parent-child links in a product taxonomy are correct without hiring annotators?
- How to test unverified knowledge-graph edges on live search traffic without hurting the user experience?
Tools and methods
- Knowledge graph with edge states — each edge is
verifiedorcandidate; only verified edges feed standard search - Exploration vs. exploitation injection — a few candidates go into lower-risk UI slots, with a per-session cap
- Behavior tracking with an anchor query — a click counts only in the context of the query that was typed
- Weighted engagement score and thresholds — an offline job promotes or prunes each edge
- Multi-armed bandit (MAB) — the article's name for the exploration strategy (elsewhere a greedy policy)
Google or a one-time email code. No password.