Problem
Lucene's BM25 scoring is term-based, uses fixed hyperparameters and cannot model nonlinear feature interactions. Yelp first added a second stage: the engine returns candidate ids, a web server fetches their features from feature stores and other services, and a standalone inference service re-ranks them. This breaks down when the candidate set or feature payload is large, because network transfer and serialization become the bottleneck.
Example from the article: Yelp's business search, reviews search, ad delivery, photo search and Home Feed all run on Nrtsearch and need ML ranking.
Solution in brief
Yelp built an Inference Plugin that loads ML models into the replica nodes of its search engine, Nrtsearch. The engine recalls documents, extracts features from the index documents or request parameters and scores each document in the same JVM. No candidates or feature vectors cross the network.
Questions this case answers
- How to score a large candidate set with a learned model within a tight latency budget when shipping candidates and their features to a separate scoring service is too expensive
- How to roll out and update ranking models in a stateful serving cluster without downtime and with a safe rollback path
- How to run ML ranking inside the search engine instead of a separate scoring service?
- How to avoid network and payload overhead when a ranking model has to score a large candidate set?
Tools and methods
- Nrtsearch Inference Plugin — in-engine scorer, callable from a Function Score query and from the rescoring phase
- MLeap models (XGBoost, neural networks) — model format run by the in-house model server inside the engine JVM
- MLflow — registry where ML engineers store trained models
- Config manager — maps each cluster to the model aliases and run IDs it must load
- Jenkins — rolls a model update across replicas one node at a time
- Prometheus — exposes model latency and error metrics for alerts
- Canary rollout — canary node before the cluster rollout of scorer changes
Google or a one-time email code. No password.