Product Applications of Yandex VLM

Ekaterina Glazkova24 May 2025

Problem

A product team gets a good general-purpose VLM and finds that each surface wants something different from it. One needs a dry, detailed answer with no invention, another fairy tales, a third a short and fast reply. Some can afford a search call, others cannot. The hardest case is multimodal search: the user uploads a photo and asks a vague question, and the text alone sends search to the wrong object.

Example from the article: a photo of an avocado with the question "How to care for it?". Text search alone would return care tips for an arbitrary plant. The query has to become "How to care for an avocado?".

Solution in brief

Add a product alignment stage after the base VLM is ready, choosing among prompt engineering, RAG, PEFT, SFT and RL by data volume and serving limits. For multimodal search, split the pipeline into image search, a VLM rephraser, text search and a VLM answer generator, and compare versions by side-by-side (SBS) assessor votes.

Questions this case answers
  • How to adapt a general vision-language model to several product tasks that differ in required objectivity, latency budget and whether external search is allowed
  • How to rephrase a vague question about an uploaded image into a precise search query, using the image as context
  • How to adapt a vision-language model to a specific product task
  • How to answer a user's question about an uploaded photo with web search and source links

Tools and methods

  • Product alignment — an extra training stage on task-specific data
  • Prompt engineering, RAG, PEFT, SFT, RL — adaptation methods, often combined
  • VLM rephraser — turns image, tags, OCR text and user words into search queries
  • VLM generator — writes the final answer from image and sources
  • SBS (side-by-side) — assessors pick the better of two answers by a criteria instruction
Continue reading — it's free

Google or a one-time email code. No password.