AI-Generated Music: Quality Evaluation Methodologies and Generation Optimization

Maksim Smolyakov24 May 2025

Problem

A Sber team building a music generator must judge whether a track is good and whether it matches the text prompt. Distribution metrics such as FAD say little about musicality and miss subtle artifacts. Human rating is the reference but is subjective, costly and hard to scale. The usual shortcut, a text-audio similarity score, can mislead, and tuning the generator's guidance coefficient by grid search costs weeks of GPU time.

Example from the article: for the prompt "Epic orchestral music for a movie trailer", a complex, dynamic generation scored low similarity, while a loud, monotonous one full of timpani and trumpets with artifacts scored high.

Solution in brief

Combine automatic checks with human rating, and treat text-audio similarity from a contrastive encoder as a rough signal, never as a quality filter. To avoid the guidance sweep, train the model with the guidance effect built in, so inference needs one pass. This is early work: instrumental music only, quality below the guided baseline.

Questions this case answers
  • How to evaluate whether a generated sample matches its text prompt and is good when automatic distribution metrics and text-audio similarity correlate weakly with human ratings
  • How to avoid tuning the guidance strength of a generator by a grid search that costs weeks of compute
  • How to evaluate whether generated content matches the prompt when automatic metrics correlate weakly with human judgment
  • How to choose between automatic metrics (FAD, CLAP-style similarity) and human rating (MOS, side-by-side) for generative output

Tools and methods

  • FAD (Fréchet Audio Distance) — compares embedding distributions of generated and reference audio.
  • MOS and side-by-side (SBS) rating — human scoring and pairwise comparison, crowdsourced via Yandex.Tasks.
  • CLAP, Clamp3, MuQ-MuLan — contrastive audio-text encoders that give a prompt-similarity score.
  • Classifier-free guidance (CFG) and cfg_scale — strength of prompt conditioning at inference.
  • Guidance-free flow-matching training — moves the CFG effect into the loss.
Continue reading — it's free

Google or a one-time email code. No password.