Problem
A credit scoring team has a tuned CatBoost/LightGBM model. Boosting is fast and interpretable, but it needs heavy feature preparation (one-hot and target encoding, features derived from text) and reacts poorly when the data domain shifts, for example under a fraud attack. It solves only one task, so client clustering or data for other teams needs separate models. A client's past loans fit boosting only as lag columns or aggregates.
Example from the article: on internal data with a baseline score around 0.63 (most likely AUC), CatBoost and LightGBM train in about 6 minutes. A single TabBERT, trained 15 hours on two GPUs, beat CatBoost by 0.001.
Solution in brief
The speaker's team tested three neural families against boosting: TabAttention, a bagging supervised autoencoder, and TabBERT/TabGPT, which treat a client's loan history as a sequence of sequences. Only TabBERT beat CatBoost as a single model, and only slightly. The practical gain comes from blending boosting with a neural model, which adds 0.005 or more.
Questions this case answers
- How to beat a tuned gradient boosting model on binary outcome prediction over tabular records by modeling each entity's sequence of past records
- How to build one entity embedding from heterogeneous internal and external sources that serves both prediction and clustering
- How to improve a credit default scoring model beyond tuned gradient boosting
- How to model a client's sequence of past loans with a BERT-style neural network
Tools and methods
- TabBERT — BERT-style model pretrained by masking loan features, then a head for scoring
- TabGPT — encoder-decoder variant that generates the next loan
- TabAttention — local and global feature views joined by self-attention
- Bagging Supervised Autoencoder Classifier — N balanced subsamples, autoencoder plus SVM each
- RNN, CatBoost, LightGBM — baselines and blend components
Google or a one-time email code. No password.