Building In-Video Search

Boris Chen, Ben Klein, Jason Ge, Avneesh Saluja, Guru Tahasildar, Abhishek Soni, Juan Vimberg, Gustavo Carmo, Meenakshi Jindal, Elliot Chow, Amir Ziai, Varun Sekhri, Santiago Castro, Keila Fong, Kelli Griggs, Mallia Sherzai, Robert Mayer, Andy Yao, Vi Iyengar, Jonathan Solorzano-Hamilton, Hossein Taghavi, Ritwik Kumar6 Nov 2023

Problem

Netflix creatives build trailers, social clips and promo reels from a catalog of hundreds of thousands of movies and episodes. To use a shot such as an exploding car, an editor had to scrub through the whole film, because nobody knew whether such a shot existed or where. Object detection and action recognition did not help: general labels were too generic, and show-specific objects (the Demogorgon in Stranger Things) do not transfer between titles.

Example from the article: a trailer for The Gray Man needs a car explosion, or a Christmas reel needs every "Merry Christmas" moment across Netflix films.

Solution in brief

Train a contrastive text-video model, starting from a pretrained image-text model, so that a shot and its description land close together in one embedding space. Shot embeddings are precomputed, stored in a feature store and indexed for nearest-neighbor search. A text query is embedded with the text encoder and matched to the closest shots.

Questions this case answers
  • How to retrieve short segments of unstructured video by a free-text query when fixed label vocabularies are too coarse and differ between titles
  • How to adapt a pretrained image-text embedding model to video units with a small in-house dataset of clips paired with descriptions
  • How to find a specific visual moment (an exploding car, a character shouting a phrase) across a catalog of hundreds of thousands of titles?
  • How to search video by free-text queries when object detection and action recognition labels are too generic or too show-specific?

Tools and methods

  • Contrastive image-text learning — joint embedding space for text and visual content: objects, scenes, emotions, actions.
  • Shot-level encoder with mean pooling — one vector per shot.
  • Fine-tuning on in-house shot-description pairs — adapts the pretrained model to video.
  • Ray Train — scalable training of the model.
  • Elasticsearch — real-time nearest-neighbor queries over shot embeddings.
  • Amazon S3 — storage of segmented shots as mp4 files.
Continue reading — it's free

Google or a one-time email code. No password.