Improving Aerial Imagery Interpretation with Diffusion Networks

Sergey Zaytsev24 May 2025

Problem

Nedra Digital builds segmentation models on aerial photographs for oil and gas licensed areas. Labeled data is scarce: the objects that matter are rare or can only be described in words by an expert, and a model trained in summer may fail in winter. Getting more data means re-flying a survey, and manual labeling is slow and error-prone. The team tested whether diffusion models could enrich small datasets automatically.

Example from the article: a YOLO11 model that segments water and swamps, where the base model misses swamps and cannot merge a lake and a swamp into one complex object.

Solution in brief

Train a LoRA and ControlNet adapter on about 20 triplets (image, binary mask, expert description), then generate images from masks. Because the mask is the input, the generated images need no manual labeling. The expanded dataset (×1000) is used to fine-tune the segmentation model.

Questions this case answers
  • How to augment a small, imbalanced image segmentation dataset with synthetic images generated from given masks so that no manual labeling is needed
  • How to segment rare, hard classes in images when labeled examples are scarce and conditions shift across seasons
  • How to augment a small, imbalanced image-segmentation dataset with diffusion-generated images instead of hand-labeling more
  • How to generate synthetic training images that match a given segmentation mask so no manual labeling is needed

Tools and methods

  • Stable Diffusion XL (SDXL) with LoRA — image generator fine-tuned to the target imagery style
  • ControlNet and inpainting — force the generated image to follow a given mask
  • Flux — higher-quality generator, but with commercial licensing issues; tested, not used in the final pipeline
  • YOLO11 — segmentation model fine-tuned on the generated set
  • Qwen3-8B — rewrites poor user prompts into detailed descriptions
  • ComfyUI — environment the team used to deploy the generation pipeline
Continue reading — it's free

Google or a one-time email code. No password.