How We Trained a Content Video Encoder Without Labeled Data

Evgeny Astafurov25 May 2024

Problem

Dzen and VK recommend videos, and new videos have almost no user feedback to place them. The team needed one semantic video embedding for three jobs: item-to-item candidates for cold videos, content moderation and categorization, and detection of semantic duplicates (the same video re-uploaded by different authors). Human labels were enough only to test quality, not to train. The production encoder (VQD) was a black box trained on search logs and overfit to titles.

Example from the article: a recommendation candidate for an anchor video is found by approximate nearest-neighbor search, so videos with similar content must lie close in the embedding space.

Solution in brief

Sample up to 64 frames per video, embed each with a frozen CNN, and feed the sequence to a BERT-style transformer (FrameBert). Pretrain it without labels by masking frame embeddings and predicting their quantized indices (BEST-RQ, borrowed from speech). Then fine-tune contrastively (infoNCE) against speech-to-text of the audio. ROC AUC on human relevance labels rose from 0.78 (VQD) to 0.88.

Questions this case answers
  • How to learn a semantic embedding of variable-length sequences of frames without human labels, using masked prediction of quantized targets
  • How to detect semantic duplicates of re-uploaded content and retrieve candidates for items that have almost no feedback yet
  • How to learn semantic video embeddings without human labels
  • How to generate item-to-item candidates for cold videos with little user feedback

Tools and methods

  • FrameBert — frozen CNN frame embeddings, then a 12-layer transformer, pooling and an MLP.
  • BEST-RQ — masked-prediction pretraining over quantized frame embeddings.
  • infoNCE fine-tuning — video representation against speech-to-text plus BERT text representation.
  • CatBoost on embedding distance — offline metric on Toloka labels.
Continue reading — it's free

Google or a one-time email code. No password.