Problem
Gusto exports company data to accounting platforms. For that, an accountant maps dozens of internal Gusto categories to hundreds of external bookkeeping accounts, which takes the accountant a long time. The team wanted to suggest the mapping automatically and show suggestions in the UI, where the user accepts one or picks another account. A prompt-based LLM suggester worked, but it took 9 seconds for 8 categories and sometimes invented accounts that do not exist.
Example from the article: the LLM returned the top three accounts per category and needed 9 seconds for 8 categories.
Solution in brief
Ship an LLM prototype first, because it needs no training data and is quick to build. Reuse its evaluation set as training data for an XGBoost ranker that scores every bookkeeping account for each Gusto category. The ranker was faster, never returned a non-existent account, and users accepted its suggestions 20% more often in an A/B test.
Questions this case answers
- How to recommend a mapping from a small set of source categories to a large list of target accounts when an LLM prompt is too slow and sometimes invents targets
- How to decide when to replace a prompt-based LLM prototype with a trained ranking model
- How to choose between an LLM and gradient boosting for suggesting the best match from a long list of options?
- How to speed up and remove hallucinated answers in an LLM-based suggestion feature?
Tools and methods
- LLM prompt (GPT-4.1, GPT-5) — first model: pick the top three accounts for each category from the full account list.
- XGBoost (XGBoostRanker) — second model: ranks all candidate accounts per category and returns the top n.
- Embedding cosine similarity — feature: similarity between the Gusto category and the account name.
- String similarity metrics — extra features for the ranker.
Google or a one-time email code. No password.