Learning from Unstructured Sequences in 2026

Boris Tseitlin23 May 2026

Problem

A fintech has plenty of data but very few labels. Loan defaults, for example, are orders of magnitude rarer than active users or transactions, and the label arrives long after the decision. The inputs are also mixed: app events, transactions and external vendor data. Teams usually cope with heavy manual feature engineering for credit scoring, credit limit increases, lifetime value forecasting and fraud. The features take many people to build and keep updated.

Example from the article: Banco Plata (a bank in Latin America, mostly Mexico) must decide on loans and limit increases from sparse default labels, while blocked legitimate transactions annoy clients.

Solution in brief

Pre-train one sequence model on client event histories without labels, then use its output embedding as features for every downstream task. The talk surveys three recipes (CoLES, NuFormer, PRAGMA) that differ in how they encode events, aggregate history and set the self-supervised objective. It is a review of other people's work, with no measurements by the speaker.

Questions this case answers
  • How to learn reusable entity embeddings from raw irregular event sequences without labels and reuse them across several downstream tasks
  • How to choose between contrastive, next-event and masked-event pre-training for event histories with rare labels
  • How to learn reusable client embeddings from raw event sequences instead of hand-crafted features
  • How to build a foundation model for credit scoring, lifetime value and fraud from transactions and app events

Tools and methods

  • CoLES — contrastive learning on sub-sequences of one client's history, with an RNN aggregator
  • NuFormer — GPT-like transformer with next-token prediction over a flat token sequence of transactions
  • PRAGMA — BERT-like masked pre-training with a hierarchical encoder, scaled to 1 billion parameters
  • CatBoost — the typical downstream model on hand-made features, the baseline these approaches replace
Continue reading — it's free

Google or a one-time email code. No password.