Problem
LY Corporation (LINE) wanted a text-to-image model that draws in the company's minimalist style from a prompt alone, to cut repetitive illustration work for designers. The same fine-tuned model gives on-style or off-style images depending on seed, guidance strength and LoRA scale. Manual or grid search costs too much time and GPU, and automating it requires turning "a good image" into a number.
Example from the article: for the prompt "A woman is holding a picket sign. The sign has the words 'hold the LINE' written on it." the same model gave a style-compliant or a non-compliant image depending on the settings.
Solution in brief
Pick a good seed per prompt with an attention-based loss. Score every image by a normalized average of four metrics (CLIP score, VQA score, HPS-V2, PickScore). Run Bayesian optimization over the guidance scales and LoRA scale with that score as the objective: 13 images per prompt and seed replace the grid.
Questions this case answers
- How to tune the settings of a generative model automatically against a composite quality score when each evaluation is an expensive black-box call
- How to merge several quality metrics with different scales and distributions into a single objective for automated search
- How to pick generation hyperparameters automatically instead of a manual grid search when image quality must be scored by a metric?
- How to choose a good starting seed for a text-to-image model without generating a full image per candidate?
Tools and methods
- Bayesian optimization — finds good settings in 13 evaluations by modelling the score surface.
- Seed screening by attention loss — ranks random seeds before the search.
- Composite image score — CLIP and VQA scores for prompt fidelity, HPS-V2 and PickScore for human preference.
- Stable Diffusion 3.5 with LoRA — the tuned generator.
- CLIP — embedding model behind the CLIP score.
Google or a one-time email code. No password.
