Problem
A medical AI company grew in three years from one production system (mammography) to five systems, with about two million studies processed. Data came in many shapes: single images, several projections per study, up to 300 3D slices, and boxes, masks or points as annotations. Clinics also changed formats.
The first setup, one code wrapper per dataset, broke as datasets, versions and in-house annotators multiplied. Folders on S3 were hard to navigate, and analytics meant reading every file and paying for each read.
Example from the article: with 10 datasets and 10 wrappers, renaming one class meant changing code in 10 places.
Solution in brief
Keep heavy DICOM files in S3 folders, describe them in one PostgreSQL schema (patient, study, image, annotation, annotator, clinic), and fill both only through Airflow pipelines. A local server with NFS and an SSD cache serves training. Everything runs in Kubernetes, so new data products such as dashboards are easy to deploy.
Questions this case answers
- How to organize storage, versioning and access for large unstructured datasets that many ML experiments and teams share
- How to automate sending data to external annotators in batches and check annotation quality before the data reaches training
- How to move from one data wrapper per dataset to a shared warehouse for medical imaging ML teams?
- How to speed up training on a local GPU server when many experiments read heavy files from a disk at once?
Tools and methods
- Amazon S3 — storage of heavy DICOM files in a folder structure.
- PostgreSQL — structured description of studies, images, annotations and pipeline status.
- Apache Airflow — ETL from clinics, annotation service and open datasets; batch annotation pipelines.
- Kubernetes — deployment of the platform and of data products.
- Supervisely — annotation service; its format is exported by pipelines.
- DVC — tried for versioning and dropped.
- ClearML — experiment tracking; its data is planned to be connected to the platform.
Google or a one-time email code. No password.
