Problem
A team builds an algorithm that produces images (MRI reconstruction, artifact removal, denoising) and needs a number that says whether the output is good. Radiologists are the true judges, but their answers differ and a doctor often cannot say if an image is "good". Dozens of automatic metrics exist, most tested only on natural photos, and it is unclear which agree with experts on medical images. A wrong pick, such as the habitual PSNR or SSIM, steers model selection toward blurry results that experts dislike.
Example from the article: a study at Philips Research and Skoltech compared 27 metrics with 14,700 ratings from 7 radiologists on brain and knee MRI.
Solution in brief
Validate candidate metrics against experts directly. Build pairs of a reference and a reconstruction from real enhancement models, have experts rate each pair on a few concrete criteria, convert the ratings to numbers and rank the metrics by Spearman correlation with them. PSNR and SSIM lost to newer metrics.
Questions this case answers
- How to choose an automatic quality metric for model outputs when it must agree with subjective expert judgment and the standard metrics were designed for a different domain
- How to build a small expert-rated benchmark and rank candidate metrics by their rank correlation with expert scores
- How to choose an automatic quality metric for generated or enhanced images?
- How to check that an offline metric agrees with what domain experts consider good?
Tools and methods
- Expert pairwise rating — radiologists compare a reconstruction with its reference on three criteria
- Label Studio — open-source annotation tool used for the ratings
- Cohen's kappa — agreement of each radiologist with their own repeated ratings
- Spearman correlation (SROCC) — ranks metrics by agreement with experts
- PIQ (PyTorch Image Quality) — library with more than 20 verified metric implementations and matching losses
- fastMRI — raw k-space data used to simulate realistic degradations
Google or a one-time email code. No password.