Analyzing Experiment Design Validity in ECG Diagnosis with Machine Learning

Ksenia Purtova25 May 2024

Problem

A team builds a classifier on data where one patient (or user) contributes many samples, for example heartbeats cut from long ECG recordings. If samples of the same person land in both the training and the test set, offline metrics come out artificially high. The model then gives bad results later, in production on real data. The same trap hits anyone who copies a published method with impressive numbers and cannot reproduce them.

Example from the article: a paper on atrial fibrillation reports very high metrics on a database of 23 patients. The author reproduced them with 10-fold stratified cross-validation (accuracy almost 100%), then regrouped the data by patient. The metrics dropped significantly.

Solution in brief

Put all samples of one patient into the same fold: split by group, not by sample. Keep the model, loss and preprocessing fixed and compare the two splits; the gap between them measures how much the leakage inflated the metrics.

Questions this case answers
  • How to split repeated samples from the same subject into train and test sets so that offline metrics are not inflated by leakage
  • How to check whether a published model's high metrics come from a flawed train/test split by re-running it under a subject-level split
  • How to split data by patient or group to avoid leakage and inflated offline metrics
  • How to check whether a published model's high metrics come from a flawed train/test split

Tools and methods

  • Stratified cross-validation (10 folds) — splits by sample and preserves class proportions; the leaky baseline.
  • Group-aware standard and stratified cross-validation — all complexes from one patient's record go to one fold.
  • Focal loss — down-weights well-classified examples to handle imbalanced classes.
  • CNN + LSTM classifier — 13 hidden layers, a single heartbeat as input, four output classes.
Continue reading — it's free

Google or a one-time email code. No password.