Crowdsourced taxonomy verification: A feedback-driven framework for refining knowledge graph relationships via online search interactions

Junpeng Niu30 Jul 2026

Problem

Search at delivery and retail apps relies on a knowledge graph (KG) of dishes, products and merchants for query expansion and ranking. LLMs and scrapers add edges from text co-occurrence, so wrong parent-child links appear. Annotators cannot keep up with millions of entities that change daily. A wrong edge makes query expansion return irrelevant items, which lowers click-through rate (CTR) and user trust.

Example from the article: an LLM links "Pho" to "Italian Noodle Soup" because the words look alike, and a search for that parent then shows irrelevant dishes.

Solution in brief

Treat every unverified edge as a hypothesis. Show it to real users in a low-risk search slot next to verified results, log the impression and the user's reaction, and score each edge by weighted engagement per impression. Edges above a threshold become verified, edges below a second threshold are deleted.

Questions this case answers
  • How to validate machine-generated relations between catalog entities continuously, when manual review does not scale and offline ground truth does not exist
  • How to collect user feedback on unproven recommendations at a bounded cost to the user experience
  • How to check whether LLM-generated parent-child links in a product taxonomy are correct without hiring annotators?
  • How to test unverified knowledge-graph edges on live search traffic without hurting the user experience?

Tools and methods

  • Knowledge graph with edge states — each edge is verified or candidate; only verified edges feed standard search
  • Exploration vs. exploitation injection — a few candidates go into lower-risk UI slots, with a per-session cap
  • Behavior tracking with an anchor query — a click counts only in the context of the query that was typed
  • Weighted engagement score and thresholds — an offline job promotes or prunes each edge
  • Multi-armed bandit (MAB) — the article's name for the exploration strategy (elsewhere a greedy policy)
Continue reading — it's free

Google or a one-time email code. No password.