Problem
OCRV, an IT provider for Russian railways, needs to know in advance which deviations will appear at a specific railway switch. Each switch gets a monthly commission inspection (KMO): 43 geometric measurements plus a text record of maintenance-standard violations. Deviations fall into three classes (planned, priority, urgent), and the urgent ones can lead to derailments or collisions. A default CatBoost model reached accuracy 0.52 but found only 14% of urgent deviations, so it was useless for the real goal. Urgent is a minority class, so the model prefers the frequent ones.
Example from the article: a model trained on 133,146 inspection records, with each switch described by measurements, passport attributes and earlier deviation texts, must flag the switch that will get an urgent deviation.
Solution in brief
Rebalance the training set so that the urgent class becomes the largest, add text embeddings and aggregate features, then lower the decision threshold for the urgent class. Recall for urgent deviations rose from 0.14 to 0.90 at the price of precision 0.23. A SHAP summary passed to an LLM explains each prediction in words.
Questions this case answers
- How to predict the most severe rare deviation class in multiclass tabular data from periodic measurements and text notes, when missing it costs far more than a false alarm
- How to choose a decision threshold and rebalance training so that recall of a minority critical class reaches a target, with predictions explained in plain language
- How to predict the most severe equipment deviations from periodic inspection measurements and text notes
- How to raise recall of a rare critical class in multiclass classification with CatBoost
Tools and methods
- CatBoost — multiclass classifier on measurements, categorical passport data and text vectors
- Random sampling, SMOTE, ADASYN — class rebalancing candidates
- Word2Vec (Russian National Corpus + Wikipedia) — 300-dimensional vectors of deviation texts
- SHAP — feature influence per prediction
- Llama 3 — turns SHAP and nearest words into a text explanation
Google or a one-time email code. No password.