How RecSys & LLMs Will Converge: Architecture of Hybrid RecoAgents

Flavian Vasile, Martin Bompaire14 Oct 2025

Problem

A deep, performance-tuned recommender wins on KPIs such as conversion rate and return on investment, but it returns ranked lists without context and cannot explain why an item was chosen. LLMs can reason over constraints, ask clarifying questions and explain, so teams want them in the loop. Fine-tuning an LLM as the recommender itself is not deployable today at large scale, for three reasons: implicit behavioural feedback is scarce and biased, catalogs of over 100M items keep changing, and real-time inference misses millisecond latency budgets.

Example from the article: a shopper asks for a gift for a 13-year-old boy under $50, and the system must turn that into product searches, retrieve candidates and explain the final picks.

Solution in brief

Keep the recommender as the retrieval and ranking backbone and wrap it in a three-stage LLM loop. The LLM turns the user's need into product queries, the recommender retrieves performance-optimised candidates, and the LLM selects among them, applies constraints and writes explanations. This is retrieval-augmented generation (RAG) with the recommender as the retriever.

Questions this case answers
  • How to combine a large-scale, latency-bound behavioural ranking system with a language model that can parse intent, apply constraints and explain choices
  • How to decide between replacing a specialised ranking model with a general sequence model and wrapping the specialised model as a tool
  • How to add a conversational LLM layer to a production recommender without losing its performance on business KPIs?
  • How to turn a free-text shopping request into queries that a performance-tuned recommender can serve?

Tools and methods

  • Two-stage recommender — recommender retrieves and ranks, LLM sits on top
  • RAG — recommender candidates ground the LLM output
  • Query decomposition — fine-tuned LLM turns intent into product queries
  • LLM re-ranking — selects and explains items from the retrieved set
  • Tool calling — the LLM calls the recommender as a tool
Continue reading — it's free

Google or a one-time email code. No password.