Problem
Netflix editors sift through hundreds or thousands of hours of raw footage per season to find story moments. Separate models tag characters, scenes and dialogue, and each emits labels or embedding vectors over its own, overlapping time ranges. A query such as "Joey in the kitchen" needs these timelines aligned, and a 2,000-hour archive holds over 216 million frames, so billions of records after several models. A ten-second delay breaks an editor's flow, and continuous shots produce thousands of near-identical candidates.
Example from the article: the character "Joey" is detected from second 2 to 8 and the scene "kitchen" from second 4 to 9. The search returns the overlap, seconds 4 to 8.
Solution in brief
Raw model outputs go to a store, an offline job cuts them into one-second buckets and merges all annotations that share a bucket, and the merged buckets are indexed in Elasticsearch as nested documents. At query time the search service builds a structured plan (filters plus vectors) and the response is rebuilt into real time ranges, as either a union or an intersection of the matches.
Questions this case answers
- How to answer a query that combines labels and embeddings from several independent models whose outputs cover different, overlapping time ranges, within sub-second latency
- How to return results at the natural boundaries of an event when the index stores fixed-size intervals
- How to find video moments that match a combination of character, scene and dialogue queries?
- How to combine outputs of several specialized models with different time spans into one searchable index?
Tools and methods
- Apache Cassandra — stores raw annotations and the fused one-second records.
- Apache Kafka — events that trigger offline fusion and then indexing.
- Elasticsearch — real-time search over fused buckets: nested documents, k-NN and ANN vector search, text matching, aggregations.
Google or a one-time email code. No password.