Post-Training Generative Recommenders with Advantage-Weighted Supervised Finetuning

Keertana Chidambaram, Qiuling Xu, Ko-Jen Hsiao, Moumita Bhattacharya25 Oct 2025

Problem

A generative recommender is pretrained to imitate the next event in a user's history. Imitation copies trends and outside suggestions as well as real preference: a user who tries a popular show and says it did not fit still teaches the model to recommend similar shows. Ratings and watch time could correct this, but PPO and DPO need counterfactual feedback, and nobody can ask users to rate hundreds of unseen titles.

The reward is also weak: watch time mixes satisfaction with free time, and over-trusting it hurts quality.

Example from the article: a reward model trained on a 1–5 proxy reward did not clearly beat two simple averages.

Solution in brief

Advantage-Weighted SFT (A-SFT) keeps the supervised next-event loss and reweights each example by its advantage from the reward model. Individual reward estimates are noisy, but they still separate high-reward from low-reward events, and that direction is what A-SFT uses. It needs no logging-policy estimate (no IPS), and a tunable parameter limits drift from the logged behavior.

Questions this case answers
  • How to improve a sequence model trained by imitation using a noisy reward signal when counterfactual feedback is unavailable and the logging policy is unknown
  • How to post-train a generative recommender on user feedback when the reward model is noisy?
  • How to improve a next-event recommender beyond imitating observed behavior without a live RL environment?
  • How to do offline policy improvement when the logging policy is unknown and IPS is too high-variance?

Tools and methods

  • A-SFT — supervised finetuning with advantage-based loss weights, the proposed method
  • HSTU — generative recommender architecture used in the study
  • Reward model — shallow head predicting a 1–5 reward for the latest title
  • Baselines — reward-weighted behavior cloning, PPO, rejection-sampling DPO/IPO, CQL
Continue reading — it's free

Google or a one-time email code. No password.