Problem
Netflix creatives build trailers, social clips and promo reels from a catalog of hundreds of thousands of movies and episodes. To use a shot such as an exploding car, an editor had to scrub through the whole film, because nobody knew whether such a shot existed or where. Object detection and action recognition did not help: general labels were too generic, and show-specific objects (the Demogorgon in Stranger Things) do not transfer between titles.
Example from the article: a trailer for The Gray Man needs a car explosion, or a Christmas reel needs every "Merry Christmas" moment across Netflix films.
Solution in brief
Train a contrastive text-video model, starting from a pretrained image-text model, so that a shot and its description land close together in one embedding space. Shot embeddings are precomputed, stored in a feature store and indexed for nearest-neighbor search. A text query is embedded with the text encoder and matched to the closest shots.
Questions this case answers
- How to retrieve short segments of unstructured video by a free-text query when fixed label vocabularies are too coarse and differ between titles
- How to adapt a pretrained image-text embedding model to video units with a small in-house dataset of clips paired with descriptions
- How to find a specific visual moment (an exploding car, a character shouting a phrase) across a catalog of hundreds of thousands of titles?
- How to search video by free-text queries when object detection and action recognition labels are too generic or too show-specific?
Tools and methods
- Contrastive image-text learning — joint embedding space for text and visual content: objects, scenes, emotions, actions.
- Shot-level encoder with mean pooling — one vector per shot.
- Fine-tuning on in-house shot-description pairs — adapts the pretrained model to video.
- Ray Train — scalable training of the model.
- Elasticsearch — real-time nearest-neighbor queries over shot embeddings.
- Amazon S3 — storage of segmented shots as mp4 files.
Google or a one-time email code. No password.