Problem
Dzen and VK recommend videos, and new videos have almost no user feedback to place them. The team needed one semantic video embedding for three jobs: item-to-item candidates for cold videos, content moderation and categorization, and detection of semantic duplicates (the same video re-uploaded by different authors). Human labels were enough only to test quality, not to train. The production encoder (VQD) was a black box trained on search logs and overfit to titles.
Example from the article: a recommendation candidate for an anchor video is found by approximate nearest-neighbor search, so videos with similar content must lie close in the embedding space.
Solution in brief
Sample up to 64 frames per video, embed each with a frozen CNN, and feed the sequence to a BERT-style transformer (FrameBert). Pretrain it without labels by masking frame embeddings and predicting their quantized indices (BEST-RQ, borrowed from speech). Then fine-tune contrastively (infoNCE) against speech-to-text of the audio. ROC AUC on human relevance labels rose from 0.78 (VQD) to 0.88.
Questions this case answers
- How to learn a semantic embedding of variable-length sequences of frames without human labels, using masked prediction of quantized targets
- How to detect semantic duplicates of re-uploaded content and retrieve candidates for items that have almost no feedback yet
- How to learn semantic video embeddings without human labels
- How to generate item-to-item candidates for cold videos with little user feedback
Tools and methods
- FrameBert — frozen CNN frame embeddings, then a 12-layer transformer, pooling and an MLP.
- BEST-RQ — masked-prediction pretraining over quantized frame embeddings.
- infoNCE fine-tuning — video representation against speech-to-text plus BERT text representation.
- CatBoost on embedding distance — offline metric on Toloka labels.
Google or a one-time email code. No password.