Creating a Semi-Synthetic Dataset to Fine-Tune a Job Posting Summarization Model with RLHF

Nikita Venediktov27 May 2023

Problem

Rabota.ru wanted a short summary on each job vacancy, hoping it would raise conversion and attract job seekers. A summarizer trained on news and articles misses what matters in a vacancy. Employers write mostly about the company, so the model summarizes the company's history instead of duties and conditions. The first attempt skipped supervised fine-tuning (SFT) and went straight to a reward model. Annotators almost never found a good output among the generations and wrote their own summary every time. Writing references by hand is slow and expensive for a team of middle and senior engineers.

Example from the article: for "Sber is looking for a janitor", most of the text describes Sber, and the stock model returns a summary about Sber.

Solution in brief

Build the SFT dataset automatically. Run extractive summarization (TextRank, LSA) to pick three sentences per vacancy, and paraphrase the results that exceed the 200-character limit. Fine-tune on the result, then collect RLHF preferences from employers who choose between two generated variants.

Questions this case answers
  • How to build a supervised fine-tuning dataset for a text-shortening generator without annotators by extracting sentences from the source under a length limit
  • How to collect preference labels for a generative model from the product's own user choices when raters cannot pick the best of several generations
  • How to build an SFT dataset for a domain-specific summarization model without annotators
  • How to generate short job-posting summaries that fit a hard character limit

Tools and methods

  • TextRank / LSA extractive summarizers — pick the top 3 sentences from a vacancy, no model training.
  • Paraphrasing model (Hugging Face) — shortens 200–300 character extracts to under 200.
  • BART / T5 — base summarization models for SFT.
  • Reward model with in-product choice — employer picks the better of two variants.
Continue reading — it's free

Google or a one-time email code. No password.