Problem
LY Corporation's Applied ML Dev team ships a background person removal (BPR) feature: it masks people who are not the photo's subject and fills the hole with an inpainting model. Papers report scores at a fixed resolution (for example 256x256) on their own masks, so models that look equal there give very different results on high-resolution photos with large masks. Inpainting has no single correct answer to compare against, and rating every candidate by hand is slow.
Example from the article: on one photo all three tested models (LaMa, HINT, FLUX.1-Fill-dev) gave a clean fill, and on another photo with a large mask their results differed strongly.
Solution in brief
Rate a small set of your own inpainting outputs by hand on a 1 to 5 scale. Compute several automatic metrics on the same outputs and keep the one with the highest Pearson correlation with the human scores. The best metric differed between the two datasets tested, so repeat the check on your data.
Questions this case answers
- How to select an automatic quality metric for a generative model when outputs have no single correct answer and human rating is too expensive to run at scale
- How to choose an inpainting model when paper metrics do not predict results on our own photos?
- How to find an automatic image quality metric that agrees with human ratings?
- How to evaluate generated images when there is no prompt and no single correct answer?
Tools and methods
- Pearson correlation with human scores — picks the metric closest to human raters
- LAION aesthetics score-v2 — single-image quality score, works on small sets
- CMMD and FID — distribution distance between originals and results, need large sets
- Phi-3.5-vision-instruct — writes prompts for prompt-based metrics
- FLUX.1-Fill-dev — diffusion inpainting model, top aesthetics score among the shown examples
Google or a one-time email code. No password.
