Shazam for Full Names: Finding a Match When a Name Can Be Written 20 Different Ways

Andrey Miroshnikov25 May 2024

Problem

A money-transfer company must check every transaction against a regulator's list of undesirable persons (terrorists, fraudsters, criminals). The list had about 32 thousand people, each with surname, name and patronymic and a date of birth in only 20% of records. One name appears in many forms: Latin spellings of "Andrey" ending in "i", "y" or a doubled letter, reordered parts, typos, short forms (Sasha for Alexander), a missing patronymic. The old system generated spelling variants by hand-written rules, and every new rule doubled the number of database queries. It missed typos, short forms and missing parts.

Example from the article: the list holds "Ivanov Ivan Ivanovich", the transaction says "Ivanov Ivan", and the transfer must still be blocked.

Solution in brief

Turn each name into a character n-gram vector, run approximate nearest neighbor search in Elasticsearch, and cut weak hits with a distance threshold. A fine-tuned multilingual-e5-small encoder then works as a second filter that removes false positives and allows a looser bigram threshold. On the business team's test set, this raised recall from 51% to 84% and cut false positives from 2.1% to 1.4%.

Questions this case answers
  • How to match a query string against a large watch list when spelling variants, typos, reordered parts and missing fields must still match, under low latency
  • How to cut false positives of an approximate match service while keeping recall, by adding a second learned similarity filter on the candidates
  • How to match names with spelling variants, typos and word permutations against a large watch list
  • How to run low-latency vector search over 15 million vectors in Elasticsearch

Tools and methods

  • Character unigram and bigram counts — vectors without training
  • Elasticsearch ANN search (HNSW) — search over 13-15 million vectors
  • multilingual-e5-small, triplet loss — second-stage encoder
  • Augmentex, Beider-Morse, own augmentations — synthetic training pairs
Continue reading — it's free

Google or a one-time email code. No password.