Gamifying Data Collection for ML in Internal Services

Vladislav Savinov27 May 2023

Problem

Yandex's internal search team ranks documents from more than 100 internal services (task tracker, documentation, personal pages) for thousands of employees a day. A CatBoost ranker is trained on clicks and ratings, but it fails on queries like "how to deploy a new service to production", where the query and the document share no root words. Fixing this needs semantic models and much more labeled data. Traffic is small, and employees in a hurry rarely rate results. Professional assessors do not fit: queries are technical, and every assessor would need access rights and agreements.

Example from the article: in a year of asking, the team got only 15 thousand ratings and under 100 suggested answers, and volunteers labeled fewer than 100 examples.

Solution in brief

The team turned data collection into a game: a "Hall of Fame" with points for ratings, suggested answers and annotated examples, a monthly top on the search page, banners and written annotation guidelines. The collected labels were then used to build a simple target from whole search sessions.

Questions this case answers
  • How to collect relevance labels for a ranking model when users rarely give feedback and experts cannot access the documents
  • How to build a ranking training target from whole user sessions instead of single clicks
  • How to collect relevance labels for internal search when users rarely leave feedback
  • How to get training data for a ranking model when assessors are too expensive or have no access to the documents

Tools and methods

  • Gamification platform ("Hall of Fame") — points, monthly top list, calls to action in search results
  • Annotation guidelines — written after the team labeled a sample itself
  • Session-level labeling — was the query resolved, and at which position
  • Linear regression over session signals — an interpretable ranking target
  • CatBoost — ranker trained on the new target
Continue reading — it's free

Google or a one-time email code. No password.