Problem
Teams that ship image generation need to know whether one model or prompt setup beats another. Classification, detection and segmentation have a ground truth, so accuracy, mAP or mIoU work. A generated image has no single correct answer: many different pictures satisfy a prompt, and some prompts have no real example at all. Human review is subjective, slow and costly, and it gives no consistent numbers for comparing models.
Example from the article: two images generated for "Draw a man holding a tennis racket in LINE style" differ in details (one man holds two rackets), and it is unclear which is better because preference varies by person.
Solution in brief
Use proxy scores from pretrained models instead of comparing with ground truth. Evaluate separately the visual quality of the image, its alignment with the prompt, and, if needed, both at once through a model trained on human preference pairs. Pick the metric by what is available: a reference image, a set of real images, or only the image and its prompt.
Questions this case answers
- How to score the quality of generated outputs automatically when no ground-truth answer exists and human judgment is subjective
- How to check that a generated output follows the instruction it was produced from, using a model-based scorer
- How to measure the quality of generated images when there is no single correct answer to compare with?
- How to check that a generated image follows the prompt it was made from?
Tools and methods
- FID, IS — distribution-level quality scores for a set of generated images
- PSNR, SSIM, LPIPS — similarity to a reference image
- Aesthetic score, CLIPIQA, Q-ALIGN — per-image quality without a reference
- CLIPScore, VQAScore, Gecko — prompt alignment
- HPS v2, PickScore — models trained on human preferences, covering quality and alignment together
- CLIP, DINO — embedders underlying most of these metrics
Google or a one-time email code. No password.
