Problem
A content service with both articles and videos wants to show a video for an article and an article for a video. The two types share no features, and nobody has labeled article-video pairs. A wrong match shows the reader irrelevant content. The talk covers Dzen, Yandex's blog service. Model inputs are the article title, URL and first sentences, or the video frames, title and recognized speech.
Example from the article: a user reads an article about planting raspberries, then watches a video of kittens. The service should offer a video about planting raspberries next to the first and articles about kittens next to the second.
Solution in brief
A three-tower model (video, query, BERT article) maps all three into one embedding space. Training uses only search logs: queries link videos and documents, so no labeled article-video pairs are needed. A small fine-tuning set from the target service adds a large gain.
Questions this case answers
- How to retrieve items of one content modality for an item of another modality when no labeled cross-modal pairs exist and only query logs are available
- How to evaluate a retrieval model across modalities with assessor judgments on top-k candidates
- How to recommend videos for an article and articles for a video when the two content types share no features?
- How to train a cross-modal retrieval model when only query logs are available and there are no labeled article-video pairs?
Tools and methods
- Three-tower embedding model — video, query and article encoders with normalized embeddings
- Hard negative mining in a batch — negatives, since logs hold only positive pairs
- Search-log datasets (VPQ, DPQ, VQD) — training pairs from watches and clicks
- Pretrained towers — initialization from the video-by-query and document-by-query models
- Toloka — assessor verdicts on top-20 candidates for the metric
- BERT — article encoder, later also video title and speech
Google or a one-time email code. No password.
