Problem
A machine learning team at Positive Technologies had to add a malware classifier to a sandbox product built on expert-written rules. The product ships on-premise, so the team never sees client data. Experts work mostly with malicious samples, so the data they send is about 1 clean to 3 malicious, while real traffic has about 1000 clean per malware. Labels come from the same rules the model must extend, so a plain classifier just learns the rules. Release also needs 100% accuracy on the experts' reference set, and an error blocks a client's clean file.
Example from the article: the expert PoC reported near-perfect metrics on a fixed train-test split. On the daily stream they fell to 0.6-0.7.
Solution in brief
Judge the model by its false positives, because only they can be unique detections. Send them to experts for relabeling every day. Gate every release on the reference set. Ship two models in one ONNX package: a silent one that only reports telemetry, and a detect one that blocks.
Questions this case answers
- How to add an ML detector to a rule-based detection system when the labels come from the same rules the model must extend, so that only false positives carry new information
- How to validate a detector on data the team cannot see and gate each release on a reference set where a single false block is costly
- How to add an ML detector to a rule-based security product when expert labels come from the very rules the model must extend
- How to choose metrics when true positives and false negatives are already covered by rules and only false positives matter
Tools and methods
- LGBMClassifier, RandomForest, LogReg — baseline models retrained and tested daily on the stream.
- SHAP — feature contribution plots that experts read to judge a verdict.
- MLflow registries (local and prod), ONNX — validation gate and safe delivery format.
- Data Lake and historical inference service — features, SHAP values, verdicts and metadata for expert review.
Google or a one-time email code. No password.
