Integrating custom dictionaries into NLP for greater clarity

17 Feb 2022

Problem

A text-matching search engine compares characters and words, so it needs help with the variation in natural language: stop words, plurals, accents and compound words. Default dictionaries built for a whole language work for most queries. When they fail, they break relevance, which the authors call a show-stopper for a search engine. The post describes two such failures: one comes from a customer's catalog, the other from German spelling rules.

Example from the article: "down" is a normal English stop word, yet a shop selling leather, suede and down jackets cannot drop it from the query "down jackets".

Solution in brief

Keep two dictionaries per category (stop words, plurals and so on): a default one per language shipped with the product and a custom one per customer. Put every dictionary behind one interface and one data structure (a trie), then look the query word up in the custom dictionary first and fall back to the static one only if the word is not found.

Questions this case answers
  • How to let each tenant of a shared text-matching engine override language-level defaults (stop words, normalization, plurals) without forking the engine
  • How to keep text normalization from erasing meaningful distinctions between words in languages with accents or compounds
  • How to handle stop words, plurals and accents in text search without hurting relevance for some customers?
  • How to let each customer override the default language dictionaries of a multi-tenant search engine?

Tools and methods

  • Stop-word, plurals and compound-word dictionaries — reduce a query to keywords and ignore plural variants during text matching.
  • Wiktionary dump — source of the default per-language word lists.
  • Custom normalization for German — maps umlauts to two-letter forms so that different words stay distinct.
  • Trie — single data structure behind all dictionaries.
  • Per-customer dictionary with lookup priority — lets a customer override or extend the defaults.
Continue reading — it's free

Google or a one-time email code. No password.