Problem
Banks, marketplaces and clinics keep long logs of timestamped events per user: transactions, clicks, medical measurements. Teams need one fixed-size vector per user to feed classification or regression (default, age group, purchase within 7 days, mortality), and the sequences are long. Contrastive pretraining (CoLES) exists for such data and a generative alternative can be built, but each wins on different datasets, and the naive sum of the two losses was worse than both single approaches on ABank and PhysioNet.
Example from the article: on the ABank default-prediction dataset (Alfa-Bank, ODS contest), a supervised model gets ROC-AUC 0.768, contrastive 0.729, generative 0.788 and the combined MLEM 0.790.
Solution in brief
MLEM treats the contrastive and generative encoders as two modalities. Pretrain a contrastive encoder and freeze it, train a generative encoder-decoder, and add a loss that pulls the two embeddings of the same sequence together and pushes different sequences apart.
Questions this case answers
- How to learn one fixed-size embedding per entity from a long unlabeled sequence of timestamped events by combining contrastive and generative pretraining
- How to combine two self-supervised pretraining objectives for event sequences so that the combined model is not worse than either objective alone
- How to learn one embedding per user from a long sequence of events without labels
- How to combine contrastive and generative pretraining for event sequences
Tools and methods
- CoLES — contrastive pretraining on subsequences of the same user
- Generative encoder-decoder — GRU encoder, transformer decoder with cross-attention to the embedding
- Sigmoid alignment loss — links generative and frozen contrastive embeddings
- CatBoost, linear head — downstream probes on the embeddings
Google or a one-time email code. No password.
