Problem
Teams with a mature XGBoost model often test a neural network by swapping the classifier and checking whether the metric improves. That swap only shows how well another model learns from the feature table already built. The table fixes what counts as an observation, which events are aggregated and how time is represented. Information those choices dropped stays invisible to any classifier, and leaky or immature labels can make an offline comparison misleading.
Example from the article: two synthetic customers have the same 12-month revolving-tradeline summaries (mean utilization 60%, latest 50%, worst status DQ60) but different chronology. Both are current now, yet the new loan's first-payment DQ60 risk may differ.
Solution in brief
Treat the classifier swap as one of three separate experiments: change the predictor, engineer better temporal features, or learn representations from detailed records. State a hypothesis for each, compare against a strong engineered baseline under the same task, information cutoff and deployment-like splits, and judge every addition by its effect on the final prediction.
Questions this case answers
- How to decide whether a learned representation of raw event histories adds value beyond a strong engineered-feature baseline when a classifier swap shows no gain
- How to build an offline evaluation whose labels, splits and features match what will be known at deployment, when outcomes mature slowly and past decisions shaped the data
- How to test whether a learned representation beats a tuned XGBoost model for predicting loan repayment?
- How to design a fair comparison between pretrained tabular models and a gradient-boosting baseline?
Tools and methods
- XGBoost — the production baseline and the predictor that combines learned representations with existing features.
- Pretrained tabular models (TabPFN, TabICL, Google's TabFM) — in-context learners evaluated on existing feature tables without parameter updates.
- Transformer over credit-report records — learns representations that feed XGBoost.
- Time-aware validation — split dates, label maturity and grouping by customer that mirror deployment.
Google or a one-time email code. No password.