Problem
A video or article platform needs to find items similar to the one a user just liked: another episode of the show, another cartoon. Tags are too coarse for this. Single-modality models mislead: two "talking head" videos look alike by the video signal while saying different things. VK faced this on VK Video, VK Clips and Dzen, where the same item representations serve recommendations, search and moderation.
Example from the article: a talking-head video about psychology and one about networks are close in video space; only the text modalities push them apart.
Solution in brief
VK trains one multimodal encoder (Zelda) with contrastive loss on tens of millions of relevant item pairs. The pairs come from cheap heuristics, then a classifier trained on LLM or assessor labels keeps only the relevant ones. Per-modality losses are added to the pair loss. Offline ROC AUC rises 4 to 17 pp over the previous model, and the embeddings are reused for candidate generation, deduplication, diversity and clustering.
Questions this case answers
- How to learn one item embedding from several modalities so that embedding distance matches relevance when single-modality similarity is misleading
- How to collect training pairs of relevant items at scale when manual labeling is too expensive
- How to build content embeddings for video and articles so that distance reflects semantic relevance
- How to collect training pairs of relevant items without manual labeling at scale
Tools and methods
- InfoNCE loss — pulls relevant pairs and the modalities of one item together
- Text, cover, frame and video encoders (BERT, EfficientNet-B0, ViT-B, FrameBERT) — fused by an MLP into one embedding
- LLM or assessor labeling plus relevance classifier — filters candidate pairs
- K-means, LLM naming, hierarchical merge — interpretable clusters
- ROC AUC of cosine distributions — offline quality metric
Google or a one-time email code. No password.