Problem
A recommender's retrieval stage narrows millions of items to thousands in under 100 milliseconds before ranking. Meta built it as a mesh of microservices: user-tower, search-and-filter and scoring services. Network hops and serialization ate the latency budget. The user model and item index shipped on different schedules and drifted apart. ML and infrastructure engineers worked in different languages, so each idea took weeks or months to ship. This capped model complexity and the number of candidates evaluated.
Example from the article: when the user model ships v2 while the item index is still on v1, v2 user vectors query v1 item embeddings.
Solution in brief
SilverTorch ("Index as Model") turns every retrieval component into a tensor or nn.Module inside one PyTorch model: item index, eligibility filter, user tower and scoring layers. One request is one GPU forward pass, so components share memory and can be co-designed. This allows more candidates and neural reranking within the same latency budget.
Questions this case answers
- How to serve candidate retrieval, filtering and scoring under a tight latency budget when separate services add network hops and version drift
- How to widen the candidate funnel and apply learned multi-objective scoring early without exceeding the serving cost budget
- How to run candidate retrieval, eligibility filtering and reranking inside a sub-100 ms budget without a mesh of microservices?
- How to retrieve far more candidates per request and score them with a neural reranker without raising serving cost?
Tools and methods
- Index as Model — index, filters and scorers as tensors and
nn.Modules in one artifact - Bloom index filter — per-item bit signature replacing the CPU inverted index
- Fused Int8 ANN search — cluster-based search over int8 embeddings in one GPU kernel
- Neural reranking and multi-task scoring — learned layers on a wide candidate pool
- PyTorch — runtime for all modules
- FAISS — CPU and GPU baseline
Google or a one-time email code. No password.