Problem
Yandex Maps shows photos in organization cards and picks the most attractive ones for blocks such as "In brief about the place". Users often upload photos to the wrong organization, so these blocks show other places and users lose trust in the recommendations. Manual checks do not scale: at 20 seconds per photo, more than 200M photos would take over 1M person-hours. There is no training dataset, and the rules are fuzzy (a facade may house many organizations).
Example from the article: the popular "Kazan Cat" sculpture in Kazan (17 thousand ratings) showed photos of other organizations in its summary block, while the Kul Sharif mosque gallery held a photo of the cat.
Solution in brief
Cluster photo embeddings inside each organization, keep only clusters that do not repeat in other (distant) organizations, then look for lone photos of one organization that fall into a unique cluster of another. Errors are rare and single, and popular places are the usual source, so this finds them without training a model.
Questions this case answers
- How to find items uploaded to the wrong entity in a catalog of hundreds of millions of images when there are no labels and no manual review capacity
- How to tell unique clusters from generic ones among image embeddings of many entities and flag a single image that matches a cluster specific to another entity
- How to find photos uploaded to the wrong organization in a place catalog with hundreds of millions of photos
- How to clean user-generated photo galleries without training data or manual moderation
Tools and methods
- ResNet-like embeddings — 256-dimensional, L2-normalized image vectors for clustering
- Agglomerative clustering — clusters photos within one organization, cosine distance
- Adjusted mutual information (AMI) — selects clustering type and hyperparameters against hand-labeled clusters
- HNSW index — approximate nearest neighbor search over more than 10M cluster centers
- Assessor labeling — validation set and candidate review
Google or a one-time email code. No password.