From RAG to Richness: How Ramp Revamped Industry Classification

Ryne Carbone15 Jan 2025

Problem

Ramp, a fintech company, labeled its customers with a homegrown industry system built from third-party data, sales entries and customer self-reporting. Categories were sometimes plainly wrong, too generic, inconsistent for similar businesses, and not auditable. Teams also converted these labels to SIC or NAICS codes through many-to-many mappings from 100+ internal levels to thousands of codes, and one internal level could map to 50 NAICS codes. So Risk, Sales and Marketing had no shared, precise view of a customer's industry.

Example from the article: a hiring platform for small businesses was "Professional Services", a bucket that also holds law firms and dating apps. In NAICS it became "561311 - Employment Placement Agencies".

Solution in brief

Ramp moved all industry labels to six-digit NAICS codes and predicts them with an in-house RAG model. Embedding similarity shortlists NAICS codes for a business, then an LLM picks one from the shortlist in two prompts. Each stage has its own metric: acc@k for retrieval and a hierarchy-aware fuzzy accuracy for the final choice.

Questions this case answers
  • How to assign each entity one label from a large hierarchical taxonomy using free-text attributes that are sparse and unevenly filled, when the label set is too big to put into one prompt
  • How to evaluate and tune a retrieve-then-select classifier stage by stage when hierarchical labels make partially correct answers meaningful
  • How to classify customer businesses into a standard industry taxonomy such as NAICS with an in-house RAG model?
  • How to evaluate a two-stage retrieval-plus-LLM classifier when there is no ground truth and labels are hierarchical?

Tools and methods

  • Text embeddings and similarity scores — shortlist candidate NAICS codes for a business
  • LLM selection with two prompts — narrow the shortlist, then choose one code with a justification
  • acc@k and fuzzy accuracy — evaluate retrieval and final prediction separately
  • ClickHouse — stores pre-computed NAICS code embeddings and computes similarity scores
  • Apache Kafka — logs intermediate results for diagnosis
Continue reading — it's free

Google or a one-time email code. No password.