LLM-based Approach to Large-scale Item Category Classification

UchidaNaotaka31 May 2024

Problem

Mercari changed its category tree in 2024: more categories and a new hierarchy. Every past item had to move to the new tree. Over 3 billion items had been listed. The "correct new category" of an old item was unknown, so a supervised classifier had nothing to train on, and rules for so many categories were impractical.

Example from the article: classifying all items with GPT-3.5 turbo would have cost about 1 million USD and taken an estimated 1.9 years.

Solution in brief

Ask GPT-3.5 turbo to label only a sample of several million items, choosing among about 10 candidate new categories per item. Use these labels as training data for a kNN model over open-source embeddings, which then labels all items. Cost and speed come from an open embedding model, multi-GPU encoding, a CPU vector index and a cap on LLM output length.

Questions this case answers
  • How to relabel a catalog of billions of records into a new label hierarchy when no ground truth exists for it and a direct LLM pass over every record is too expensive and too slow
  • How to train a cheap classifier from a small LLM-labeled sample so that a large corpus can be labeled within a fixed compute budget
  • How to move billions of existing items to a new category tree when no labels exist in the new tree?
  • How to label a huge catalog with an LLM when calling it for every item costs too much and takes too long?

Tools and methods

  • GPT-3.5 turbo (OpenAI API) — labels the sampled items with a short reason (chain-of-thought) and a category ID
  • kNN over embeddings — takes the most frequent category among X nearest labeled items as the prediction
  • multilingual-E5-base — open embedding model, chosen over larger E5 variants for its cost/accuracy balance
  • Sentence Transformers — multi-process encoding over several GPUs
  • Voyager — vector index that is fast enough on CPU
  • LangChain — stream() keeps JSON valid when max_tokens cuts the answer
Continue reading — it's free

Google or a one-time email code. No password.