Neural Network Architectures in Credit Scoring

Yuri Bolkonsky25 May 2024

Problem

A credit scoring team has a tuned CatBoost/LightGBM model. Boosting is fast and interpretable, but it needs heavy feature preparation (one-hot and target encoding, features derived from text) and reacts poorly when the data domain shifts, for example under a fraud attack. It solves only one task, so client clustering or data for other teams needs separate models. A client's past loans fit boosting only as lag columns or aggregates.

Example from the article: on internal data with a baseline score around 0.63 (most likely AUC), CatBoost and LightGBM train in about 6 minutes. A single TabBERT, trained 15 hours on two GPUs, beat CatBoost by 0.001.

Solution in brief

The speaker's team tested three neural families against boosting: TabAttention, a bagging supervised autoencoder, and TabBERT/TabGPT, which treat a client's loan history as a sequence of sequences. Only TabBERT beat CatBoost as a single model, and only slightly. The practical gain comes from blending boosting with a neural model, which adds 0.005 or more.

Questions this case answers
  • How to beat a tuned gradient boosting model on binary outcome prediction over tabular records by modeling each entity's sequence of past records
  • How to build one entity embedding from heterogeneous internal and external sources that serves both prediction and clustering
  • How to improve a credit default scoring model beyond tuned gradient boosting
  • How to model a client's sequence of past loans with a BERT-style neural network

Tools and methods

  • TabBERT — BERT-style model pretrained by masking loan features, then a head for scoring
  • TabGPT — encoder-decoder variant that generates the next loan
  • TabAttention — local and global feature views joined by self-attention
  • Bagging Supervised Autoencoder Classifier — N balanced subsamples, autoencoder plus SVM each
  • RNN, CatBoost, LightGBM — baselines and blend components
Continue reading — it's free

Google or a one-time email code. No password.