Applying multitask learning to AI models at LinkedIn

Ji Yan, Sen Zhou, Dansong Zhang, Anastasiya Karpovich19 Aug 2022

Problem

A product usually has many ML models, one per task, each with its own owner and data. New tasks start without training data and cannot reuse what neighbouring models have learned. Merging tasks is hard when label rates differ widely, input features differ across products, and models use different architectures and losses. LinkedIn faced this in job recommendation, job search and recruiter search.

Example from the article: a job-application model and a job-view model both come from job search, but only about 10% of job views lead to an application, so views swamp the application signal.

Solution in brief

LinkedIn built a multitask training framework that lets each task keep its own dataset, features and model definition. The framework unifies the feature schemas, finds the parts of the models that overlap and trains them as one shared component, with task-specific components on top. Training runs either jointly or iteratively. Tasks are chosen by domain knowledge. Two applications gained in model quality and product metrics.

Questions this case answers
  • How to train one model for several related tasks when the tasks differ in label distribution, input features and architecture
  • How to transfer learning from a data-rich task to a data-poor task that has a similar structure but a different domain
  • How to train related ML tasks together when their labels and input features differ?
  • How to give a new task with little training data the benefit of a data-rich sibling task?

Tools and methods

  • Multi-task learning (shared-bottom model) — one shared sub-model plus task-specific sub-models, trained on several tasks at once
  • Joint training — combined batches and a combined loss back-propagated into the shared model
  • Iterative training — each task and its model trained in turn, with gradients from the shared layers
  • Translearn — LinkedIn's in-house training framework that supports iterative and joint training
Continue reading — it's free

Google or a one-time email code. No password.