Matching external companies to LinkedIn's Economic Graph at scale

Remi Mir, Tianhao Lu, Xiaoqiang Luo, Yunpeng Xu25 Apr 2022

Problem

Sales and advertising teams keep company lists in CRM exports and CSV files, with a name, often a website, and sometimes an industry and address. A platform must link each record to the right catalog page. Names collide (two firms called "Graco" in different industries and cities), branches differ from headquarters, and millions of pages cannot be scanned per record. Matches are needed almost instantly for a CRM sync and in bulk for large advertiser lists.

Example from the article: A request with the name "Bank of Remi" and the website "https://www.bankofremi.com" is resolved to the best-ranked LinkedIn organization page.

Solution in brief

LinkedIn's Organization Entity Resolution (OER) service cleans the request, pulls candidates from an inverted index keyed on name n-grams and website domains, and shortlists 10 with a cheap first-level ranker. A second-level model scores each pair on name, website, industry and location features. User overrides become training labels, and match rate covers unlabeled traffic.

Questions this case answers
  • How to link noisy structured records from external systems to a canonical entity catalog at millions-of-records scale, both in batch and with low latency
  • How to monitor the quality of a ranking service when most requests have no ground truth
  • How to match millions of external company records (name, website, industry, address) to canonical company pages?
  • How to keep an entity-matching service fast enough for online use when one request has thousands of possible candidates?

Tools and methods

  • Inverted index on name n-grams and website keys — candidate retrieval without a full scan
  • Two-level ranking (L1 string similarity, L2 feature model) — more signals on a short list, low latency
  • Wide-n-deep and wide-only models — score request-candidate pairs; the deep part handles near-synonymous industry names
  • Apache Spark — offline batch access
  • Tracking events, match rate dashboard, alerts — monitoring without ground truth
Continue reading — it's free

Google or a one-time email code. No password.