Problem
Ramp, a fintech company, labeled its customers with a homegrown industry system built from third-party data, sales entries and customer self-reporting. Categories were sometimes plainly wrong, too generic, inconsistent for similar businesses, and not auditable. Teams also converted these labels to SIC or NAICS codes through many-to-many mappings from 100+ internal levels to thousands of codes, and one internal level could map to 50 NAICS codes. So Risk, Sales and Marketing had no shared, precise view of a customer's industry.
Example from the article: a hiring platform for small businesses was "Professional Services", a bucket that also holds law firms and dating apps. In NAICS it became "561311 - Employment Placement Agencies".
Solution in brief
Ramp moved all industry labels to six-digit NAICS codes and predicts them with an in-house RAG model. Embedding similarity shortlists NAICS codes for a business, then an LLM picks one from the shortlist in two prompts. Each stage has its own metric: acc@k for retrieval and a hierarchy-aware fuzzy accuracy for the final choice.
Questions this case answers
- How to assign each entity one label from a large hierarchical taxonomy using free-text attributes that are sparse and unevenly filled, when the label set is too big to put into one prompt
- How to evaluate and tune a retrieve-then-select classifier stage by stage when hierarchical labels make partially correct answers meaningful
- How to classify customer businesses into a standard industry taxonomy such as NAICS with an in-house RAG model?
- How to evaluate a two-stage retrieval-plus-LLM classifier when there is no ground truth and labels are hierarchical?
Tools and methods
- Text embeddings and similarity scores — shortlist candidate NAICS codes for a business
- LLM selection with two prompts — narrow the shortlist, then choose one code with a justification
- acc@k and fuzzy accuracy — evaluate retrieval and final prediction separately
- ClickHouse — stores pre-computed NAICS code embeddings and computes similarity scores
- Apache Kafka — logs intermediate results for diagnosis
Google or a one-time email code. No password.