Opening the Door to the World of Machine Learning's Dark Forces: Zero-shot Classification for Ruthless Non-Personalized Recommendations

Artem Sebalo28 Oct 2023

Problem

Rabota.ru, a job board, shows non-personalized collections such as "Work in a warehouse", "Remote work" or "Work for women". Simple collections run on filters and keywords, but "Work for women" has no filter, so the team piled up rules. With 180–200 collections and a changing team, the rules became unmaintainable. "Work for women" was the most clicked tile among about 120 collections, yet near 80th by hit rate: users opened it and clicked nothing inside.

Example from the article: the rule matched "administrator" but not "1C administrator", and "sales manager" but not "sales manager for construction equipment". Generic client-facing and entry-level vacancies buried the better ones.

Solution in brief

Instead of rules, score every vacancy with a pre-trained zero-shot classifier from Hugging Face. The labels are "man" and "woman", the input is the vacancy title and, separately, its description, and the score is the difference between the two label probabilities. The two scores are combined with a higher weight on the title, and the result sorts vacancies for the collection.

Questions this case answers
  • How to rank items inside a non-personalized collection by suitability for a target audience when there is no query, no user history and no labeled data
  • How to replace hand-written keyword rules for many collections with a zero-shot classifier and tune its prompt quickly, before running an A/B test
  • How to replace hand-written rules for non-personalized collections with zero-shot classification
  • How to build a score that sorts job ads (or other items) by suitability for a target audience without labeled data

Tools and methods

  • Zero-shot classification — a pre-trained model scores candidate labels with no training or labeling.
  • Weighted title and description score — difference of label probabilities, weighted and normalized.
  • Hugging Face — hosts the model; the code is three lines.
  • ClickHouse — response and view history with visitor gender, used to check the score and tune weights.
Continue reading — it's free

Google or a one-time email code. No password.