Problem
Mercari changed its category tree in 2024: more categories and a new hierarchy. Every past item had to move to the new tree. Over 3 billion items had been listed. The "correct new category" of an old item was unknown, so a supervised classifier had nothing to train on, and rules for so many categories were impractical.
Example from the article: classifying all items with GPT-3.5 turbo would have cost about 1 million USD and taken an estimated 1.9 years.
Solution in brief
Ask GPT-3.5 turbo to label only a sample of several million items, choosing among about 10 candidate new categories per item. Use these labels as training data for a kNN model over open-source embeddings, which then labels all items. Cost and speed come from an open embedding model, multi-GPU encoding, a CPU vector index and a cap on LLM output length.
Questions this case answers
- How to relabel a catalog of billions of records into a new label hierarchy when no ground truth exists for it and a direct LLM pass over every record is too expensive and too slow
- How to train a cheap classifier from a small LLM-labeled sample so that a large corpus can be labeled within a fixed compute budget
- How to move billions of existing items to a new category tree when no labels exist in the new tree?
- How to label a huge catalog with an LLM when calling it for every item costs too much and takes too long?
Tools and methods
- GPT-3.5 turbo (OpenAI API) — labels the sampled items with a short reason (chain-of-thought) and a category ID
- kNN over embeddings — takes the most frequent category among X nearest labeled items as the prediction
- multilingual-E5-base — open embedding model, chosen over larger E5 variants for its cost/accuracy balance
- Sentence Transformers — multi-process encoding over several GPUs
- Voyager — vector index that is fast enough on CPU
- LangChain —
stream()keeps JSON valid whenmax_tokenscuts the answer
Google or a one-time email code. No password.
