Advanced Use of GigaChat at Rabota.ru: Synthetic Data and Search

Anastasiya Ovchinnikova25 May 2024

Problem

Rabota.ru, a job search service, shows each vacancy on the main page as a snippet with a text field of only 200 characters. The team used the first 200 characters of the posting, which usually hold company self-promotion, while schedule, shift work and perks sit in the middle, so candidates miss suitable jobs. Training a summarizer needs at least 30 thousand "vacancy - summary" pairs, and an annotator writes only 80-100 summaries a day (about 4 minutes each). Separately, full-text search returns nothing for synonym queries.

Example from the article: the query "bodyguard" (telokhranitel) finds nothing although there are many "security guard" (okhrannik) vacancies, and the user leaves the site.

Solution in brief

Fine-tune a small T5 model in two stages: supervised fine-tuning (SFT) on a dataset that mixes human, extractive and GigaChat-paraphrased summaries, then RLHF with a reward model. For empty search results, embed vacancy titles with GigaChat Embeddings and fall back to vector search when text search returns fewer than 5 vacancies.

Questions this case answers
  • How to train a small summarizer for short snippets when annotators can write only a few dozen targets per day
  • How to build a synthetic training set for a small seq2seq model with a large language model as the data generator, and how to fix empty search results for synonym queries without a synonym list
  • How to summarize long job postings into a short snippet without hand-labeling tens of thousands of examples
  • How to build a synthetic SFT dataset for a small seq2seq model with an LLM

Tools and methods

  • T5 — small summarization model, trained with SFT and then RLHF (PPO)
  • GigaChat — paraphrases extractive summaries into synthetic SFT data; generates vacancy texts from fields
  • TextRank — builds extractive summaries for the synthetic dataset
  • GigaChat Embeddings + Elasticsearch — vector index of vacancy titles for fallback search
Continue reading — it's free

Google or a one-time email code. No password.