Neural Network Outfit Composition

Ekaterina Glazkova27 May 2023

Problem

A fashion marketplace wants to show a shopper a full outfit built around one item: a product they are viewing or a photo they uploaded. Hiring stylists to compose outfits does not scale. Experts are few, the assortment changes daily, new items keep arriving, and outfits are needed for almost every product. Choosing each next item only by pairwise compatibility with the first one gives mismatched or monotonous outfits.

Example from the article: given a dress, the system returns five items (outerwear, shoes, bag, accessories) that form one outfit, shown as a collage or a product carousel.

Solution in brief

The team adds items one at a time. A model takes the pre-trained embeddings of the items already chosen, predicts the embedding of the next item for a required category, and picks the nearest item of that category from the index. A plain MLP on averaged embeddings matched or beat the published baselines, and a transformer did not improve on it.

Questions this case answers
  • How to build a complete set of complementary items step by step from one seed item, where each new item must match all items already chosen and not only the first one
  • How to evaluate a sequential set-completion model offline when the benchmark labels are noisy and the model must work on shared pre-trained embeddings with fast retrieval
  • How to build complete outfits from a marketplace catalog starting from one item
  • How to recommend complementary products that match each other, not just the first item

Tools and methods

  • Pre-trained product embedding network — turns every item photo into a vector; the same index serves other tasks.
  • MLP on top of averaged embeddings — predicts the embedding of the next item.
  • InfoNCE loss with in-batch negatives — trains the next-item prediction.
  • Clothing detector — crops the sold item out of a multi-item photo.
  • Fill-in-the-blank (FITB) on Polyvore and A100 — offline benchmark; expert review for generated outfits.
Continue reading — it's free

Google or a one-time email code. No password.