Snap Ads Understanding: From Pixels to Words

23 Apr 2025

Problem

An ad platform gets a large volume of new ads every day. It must check each one against policy and show it to the right users. That requires knowing what the creative shows and says: themes, products, promotions. The usual tool is one embedding per video, and it has three faults. It squeezes scene changes and on-screen text into a single vector. It cannot be read, so debugging a downstream model is hard. It carries no explicit labels, so a query like "all ads that mention 50% off" is impossible, and every new attribute means retraining.

Example from the article: a Snap Spectacles ad with a yellow "SALE" frame. Frame-averaged CLIP similarity to the keyword "Spectacles" was 0.2603, barely above the 0.2586 of a random string.

Solution in brief

Turn each creative into a structured text description. Sample frames, transcribe the audio, and ask a vision-language model (VLM) with a task-specific prompt to write one narrative of brands, products, context and events. Store the text in a relational and vector database and search it by keyword or by text embedding.

Questions this case answers
  • How to turn unstructured video and image content into explicit, queryable attributes when single-vector embeddings lose scenes and on-screen text and cannot be queried by exact concept
  • How to add new content-derived features to downstream models and rules without retraining a representation model for each one
  • How to make video and image ads searchable and reviewable for policy moderation without training a detector for every new attribute?
  • How to find all ads that mention a specific promotion, product or theme when embeddings cannot answer an exact query?

Tools and methods

  • Vision-language model (VLM) — fuses frames and transcript into one description
  • Speech-to-text model — transcribes voiceover or soundtrack (model not named)
  • Text index and vector store — keyword search and embeddings of the descriptions
  • CLIP — baseline: mean-pooled frame embeddings compared with keyword embeddings
Continue reading — it's free

Google or a one-time email code. No password.