Generation of three-dimensional synthetic data for training neural networks

Angelina Andreeva, Semen Vervein29 Oct 2022

Problem

A power supply company inspects insulators on transmission line poles. Climbing inspection is expensive because all consumers must be disconnected, so a drone photographs each pole and neural networks look for defects: cracks, chips and contamination. Instance segmentation cuts out each insulator, and a classifier labels it. Real chips and cracks are rare: among tens or hundreds of thousands of images there were about 500–1000 defects, while dirty insulators were plentiful. The classifier therefore cannot learn the rare defects.

Example from the article: cutting real cracks and pasting them on healthy insulators gave many unusable samples that annotators still had to review.

Solution in brief

Render insulators with chips and cracks from 3D models in Blender, and automate it with the Blender Python API. Rendered images alone make the classifier learn the synthetic style, so they are passed through an image-to-image domain adaptation network trained on real photos. A script then selects the good outputs without annotators.

Questions this case answers
  • How to train an image classifier for a rare defect class when real positive examples are too few and manual labeling of augmented samples is too costly
  • How to close the gap between rendered and real images so that a classifier trained on a mix does not learn the synthetic style
  • How to train a defect classifier when real defect photos are 500–1000 among hundreds of thousands of images?
  • How to generate realistic synthetic defect images from 3D models with automatic masks and bounding boxes?

Tools and methods

  • Blender — 3D models, boolean operations for chips, procedural ceramic material for cracks, 360-degree backgrounds, scripted rendering
  • Blender Python API — random camera, background, defect position and size, with image, mask and bounding box output
  • Domain adaptation (adversarial, NVIDIA network) — repaints rendered images in the style of real photos
  • ResNeSt-50 — weak classifier used to measure the effect of synthetic data
Continue reading — it's free

Google or a one-time email code. No password.