LM-based query categorization for query understanding

Kazuma Arimura22 Dec 2023

Problem

A marketplace search team needs to know which product category a query belongs to, so the engine can filter, boost or change retrieval logic. The classic answer is a map from query strings to category ids. It is easy to explain, but it needs constant care: synonyms, name conflicts and irregular cases are fixed by hand, and the rules go stale as listing trends change and new products appear. No rule set covers all queries, and the Mercari US team had run theirs for several years.

Example from the article: a rule maps the query "shoes" to category 3, and that category id is passed to the search engine as a filter.

Solution in brief

Train a classifier on query and click logs: the label of a query is the category of the items users clicked after it. The team first used a classic ML model (micro-F1 0.72), then replaced it with DistilBERT embeddings plus a trained classification layer (micro-F1 0.80). In an online test the language-model version doubled the coverage of converted keywords.

Questions this case answers
  • How to assign free-text user queries to a large hierarchical taxonomy when hand-written rules cannot cover the long tail and drift over time
  • How to train a query classifier when the only supervision is the behavior of users on the results
  • How to map a search query to a category of the product taxonomy without maintaining thousands of manual rules?
  • How to build training labels for query categorization from click logs?

Tools and methods

  • Rule-based categorization — query-to-category map; the baseline and the explainable fallback
  • Click-log classifier — multi-class model that predicts the leaf category from the query text
  • DistilBERT — pretrained language model that turns a query into an embedding for the classifier
Continue reading — it's free

Google or a one-time email code. No password.