Method

Multimodal fusion

Combining modalities by early, late or cross-attention fusion.

13 cases from 11 companies apply Multimodal fusion. 5 were checked in an online A/B test; 2 report results in production.

Typical tasks: Recommend from what the user just did, Retrieve and rank from millions of items, Match entities, Learn user and item embeddings and 8 more. Companies: Yandex, VK, Avito, Netflix, Grab, DoorDash and 5 more. Often combined with: Time-based validation, Gradient boosting, Two-tower model, Hard negative mining, Deep & Cross Network and 46 more.