Problem
Rabota.ru is a job search site. Its main page is a feed of vacancies, and the summary field is the main source of information on which a job seeker decides to click or keep scrolling. The summary was just the first 200 characters of the description. Long, rambling descriptions full of filler leave nothing useful in that cut, so people skip potentially good vacancies. In the team's user-journey research, 69% of respondents named vacancy selection as the hardest stage.
Example from the article: in a long vacancy where the important details sit at the end, the first 200 characters said nothing about the job.
Solution in brief
Generate short abstractive summaries with T5, then tune the model to the team's own notion of a good summary with RLHF. Annotators compare pairs of summaries, a reward model learns those preferences, and PPO tunes T5 against it. A preliminary A/B test showed clicks and views up by more than 5%.
Questions this case answers
- How to generate short abstractive summaries of long documents that fit a fixed character limit, tuned to an internal notion of quality without a labeled corpus of ideal summaries
- How to collect pairwise human preferences cheaply from a small team and use them to tune a generator with a reward model
- How to generate short, informative summaries of long listings that raise click-through
- How to tune a seq2seq model to a team's own idea of a good summary without a large labeled corpus
Tools and methods
- T5 (a version adapted for Russian) — the model being tuned; chosen over mBART and GPT after human review of samples
- mBART — second candidate; used with T5 to produce the answer pairs for annotation
- Reward model — trained on pairwise human labels with a ranking loss
- PPO with KL penalty (TRL, PPO Trainer) — the RLHF optimization loop
- Airflow, Postgres — generation for newly arrived vacancies and storage for the backend
Google or a one-time email code. No password.