Problem
A computer-vision team at Cels (chest CT, pelvic MRI) needs to detect pathologies that almost never appear in routine scans. With 20-30 labeled examples the class is badly imbalanced and detection quality stays unacceptable. Getting more real data was not possible in the author's case. Standard remedies are transfer learning, foundation models, few-shot learning and heavy augmentation.
Example from the article: only about 10-15 displaced rib fractures were available for a rib-fracture detection service.
Solution in brief
Use an open diffusion model to create extra examples of the rare class, with the simplest setup that works. Either teach the model the pathology with a small LoRA, or paste a few real findings onto healthy images and let the model blend them in by inpainting. Review the output by eye, because most generations are unusable.
Questions this case answers
- How to generate extra examples of a rare visual class from a few dozen real images by adapting a pretrained generative model with minimal data
- How to decide whether synthetic examples are worth building or whether collecting more real examples is cheaper
- How to generate synthetic images for a rare class when only 10-30 real examples exist
- How to add rare pathology examples to a training set with a diffusion model and minimal effort
Tools and methods
- Stable Diffusion 1.5 / Prompt2MedImage — open base model; Prompt2MedImage is Stable Diffusion 1.5 trained on medical image-text pairs
- LoRA — fine-tuning of the U-Net with a small share of added parameters
- Inpainting — regenerating a masked region of a real image
- Diffusers, MONAI — libraries for training and for medical models
- Manual review — filtering rejected generations
Google or a one-time email code. No password.
