Building the incremental and online training platform at LinkedIn

Chen Zhu, Tao Huang, Lijun Tang, Yujie A., Yang Pei, Ratnadeep Mangalvedhekar, Ming-Ju Valentine Lin, Yun Zhang, Hui Wang14 Oct 2025

Problem

LinkedIn retrained its Feed, Ads and Jobs ranking models in daily batches. Each run repeats training on data from previous days, so the model is about a day stale and a failure in a long job forces a restart. Training data prepared offline also carries feature values updated after the event was logged, so training and inference disagree. Hourly or continuous updates add new failure modes: streaming ingestion, fragile user code, lost state, silent drift, miscalibrated scores.

Example from the article: a predicted click-through rate of 0.7 should match a 70% observed click rate, which matters for Ads auctions and bidding. Batch isotonic regression takes hours, which cancels the freshness gain.

Solution in brief

Train each new model version from the previous checkpoint on only the newest data, in a managed recurring pipeline. Training data is built in a nearline stream that logs the features used at ranking time. Users submit a serialized training graph instead of code, and calibration mappings are refreshed separately and reused by online-trained models.

Questions this case answers
  • How to update a production prediction model continuously on streaming data without full retraining and without training-serving feature skew
  • How to keep model scores calibrated when the model is updated more often than a batch calibration job can run
  • How to keep ranking models fresh by training on the latest data instead of full daily retraining?
  • How to make training features match the features used at inference when training data is built offline?

Tools and methods

  • Nearline feature attribution — joins engagement and ranking-feature streams into training events.
  • Static training graph — serialized forward, loss, backward and optimizer steps, run centrally.
  • Ray-based ingestion library — parallel reading of Kafka partitions into tensors.
  • Calibration pipeline — periodically updated mappings reused by online-trained models.
  • Apache Kafka, Apache Flink, Ray, Kubernetes, MLflow, Prometheus — streaming, scale-out, scheduling, monitoring.
Continue reading — it's free

Google or a one-time email code. No password.