Leveraging Commerce Data for Outcome-Based Relevancy in Agentic Recommendation Systems

Maxime Vono, Thibault Becker, Hamlet Jesse Medina Ruiz, Otmane Sakhi5 Feb 2026

Problem

Agentic shopping assistants turn a user's text query into a list of products. Most of them rely on general text encoders or large language models. These models check whether a product description matches the query. They do not know which matching products people actually click or buy. Two products can fit the query equally well on brand, category and attributes, yet differ a lot in how often shoppers click and buy them.

Illustrative example: a query for running shoes matches dozens of models by text, and the ranker must put the ones shoppers buy near the top.

Solution in brief

Use commerce data at both stages of a two-step pipeline. Retrieval uses a small two-tower text embedding model fine-tuned on organic clicks. Re-ranking orders the retrieved candidates by a sales score, here the share of sales over the previous 7 days. Both stages are judged by the rank of the product users actually clicked or bought.

Questions this case answers
  • How to rank candidate items for a text query by observed user outcomes when text similarity alone does not separate items that match equally well
  • How to benchmark a retrieval model against zero-shot encoders using only logged clicks and purchases as ground truth
  • How to make product recommendations for a free-text query rank by purchase outcomes, not only by semantic match?
  • How to evaluate retrieval and re-ranking by the rank of the product the user actually clicked or bought?

Tools and methods

  • CLEPR (Contrastive Language Embedding for Product Retrieval) — two-tower bi-encoder of about 120M parameters on a Multilingual-MiniLM-L12-H384 backbone, fine-tuned on organic clicks.
  • PSales — re-ranking score: the proportion of sales of a product over the previous 7 days.
  • Normalized rank of the clicked or purchased product — the outcome-based metric for both stages.
Continue reading — it's free

Google or a one-time email code. No password.