When a Vision Transformer Really Came in Handy

Daniil Storonkin25 May 2024

Problem

A metallurgical company receives scrap by wagon, 50–70 tons each. Contamination of the scrap changes furnace yield and energy use, and heavily contaminated scrap costs the supplier a commission. By GOST 2787 an inspector rates contamination by eye, so the number depends on who stands at the wagon, and the person works in a hazardous zone. The team needed an automatic contamination percentage per wagon from camera photos.

Example from the article: most wagons have 1.5–4% contamination (range 0–8%), and the business cares about the critical ones above 5%. Published image-regression models did not help: the best MAE on the UTKFace benchmark is 3.7, the speaker calls it "a bit sad" next to a 0–8 value range.

Solution in brief

Photograph the wagon after every magnet grab, have several inspectors rate each unloading, and fine-tune a Swin Transformer V2 to regress the contamination percentage of each photo. The wagon value is a weighted average over its photos. Swin beat the convolutional networks in a grid search over all torchvision architectures.

Questions this case answers
  • How to estimate a continuous quantity from photos when human labels are subjective and noisy and published benchmarks cover only classification
  • How to aggregate per-photo estimates into one value for a whole batch of items when the batch is heavy and not uniform
  • How to estimate the percentage of contamination in scrap metal from camera photos
  • How to choose an image regression architecture when published benchmarks cover only classification

Tools and methods

  • Swin Transformer V2 (regression head) — per-photo contamination percentage
  • Grid search over torchvision architectures — how the model was chosen
  • YOLOv8 — vehicle and magnet detector in the capture module
  • Triton Inference Server — model serving
  • MLflow — experiment comparison, with a generated report per run
Continue reading — it's free

Google or a one-time email code. No password.