Augmenting Fuzzy Matching with Human Review to Maximize Precision and Recall

Robert B. Kaspar, Mitchell Koch18 Oct 2022

Problem

DoorDash's ad exchange lets a consumer-goods brand promote its product inside grocery stores. Brand and store are different parties, so a campaign needs every matching item found in thousands of store catalogs. Doing it by hand does not scale beyond a few dozen advertisers. Barcodes (UPC) are missing, wrong or duplicated in store data, and UPC matching found only about half of the wanted matches. Matching a competitor's product is unacceptable, because the advertiser would be billed for promoting a rival.

Example from the article: the brand "Fizzy Cola" asks to promote "Diet Fizz 12 oz. can (6-pack)", and the system must find that item across all store catalogs without ever picking a competitor's soda.

Solution in brief

A fuzzy name matcher proposes several candidates per item with a low score threshold, which keeps recall high. Human reviewers confirm each candidate, which gives near-perfect precision. Top-k candidates, a smaller search pool and metadata filters such as quantity improve the matcher and cut review effort.

Questions this case answers
  • How to link records from two sources at scale when identifiers are unreliable and wrong links are very costly
  • How to combine a high-recall automatic matcher with manual review to get near-perfect precision under a limited review budget
  • How to link advertiser products to items in thousands of store catalogs when UPC codes are missing or unreliable?
  • How to reach near-zero false matches with a fuzzy matcher when no classifier is accurate enough?

Tools and methods

  • Fuzzy name matching — word-by-word score from 0 to 100.
  • Score threshold — discards weak candidates; trades recall for precision.
  • Top-k candidates — shows reviewers 3 to 10 candidates per item instead of one.
  • Metadata and search-space limits — quantity filter, few advertisers.
  • Human review by an external vendor — at least one reviewer per candidate.
Continue reading — it's free

Google or a one-time email code. No password.