A small BERT towards Large Medical Models

Mikita Shchutski4 Jun 2022

Problem

Teams that want a medical embedding space from electronic health records (EHR) face two obstacles. The data is collected by hospitals, so the data scientist cannot control its errors. The feature space is also huge: conditions alone can reach about 20k features. Record gaps, vanishing diagnoses and inconsistent measurement units mean that "garbage in, garbage out" applies. Visit intervals do not follow a standard time-series pattern either.

Example from the article: cholesterol and hemoglobin measurements contain many zeros and clusters on a different scale, and the time between visits is dominated by "tomorrow" and "in a week" returns.

Solution in brief

Start with conditions only, clean the data with domain experts, and convert each patient history into a BERT-style token sequence: visits as concept lists with separators, as in the BEHRT paper. Add date distances and event types to the positional encodings. Grow complexity in small steps so failures stay explainable.

Questions this case answers
  • How to learn informative entity embeddings from irregular, multi-source event histories whose recording errors the modeler cannot control
  • How to detect and separate measurement-scale errors from true anomalies in noisy recorded values before modeling
  • How to turn electronic health records into token sequences that a BERT model can pretrain on?
  • How to find unit errors, zeros and other anomalies in clinical measurements before modeling?

Tools and methods

  • BEHRT (BERT for EHR) — BERT applied to visit sequences of medical concepts.
  • Masked language modeling (MLM) and next-sentence-style tasks — pre-training on record sequences.
  • Sentence-BERT — a candidate way to get a clustering-friendly embedding space.
  • Shallow anomaly detectors — a mixture of simple detectors on single features.
  • SHAP — feature importance for embedded and clustered patients.
Continue reading — it's free

Google or a one-time email code. No password.