Product ML: Expectations vs. Reality. How to Describe the Organization in Brief

Nikita Kiselev25 May 2024

Problem

Yandex Maps runs an "Ideas" mode that recommends where to go and shows a circular photo with a caption on the map, then a long description and venue features. The text is generated from user reviews by YandexGPT, and an out-of-the-box prompt did not work in production. The model refused to describe venues, invented praise ("cozy atmosphere, polite staff") and negativity, and produced captions from board-game or cocktail names. Photo circles did not match captions. Millions of venues need such texts, and editors and users judge them directly.

Example from the article: an anti-cafe got a caption about "Secret Hitler", a board game that made it stand out, and a beer restaurant showed waffles under a caption about duck.

Solution in brief

Instead of one prompt, build a chain of narrow models around the LLM, each fixing one failure found in production: filter refusals, fix style, remove filler and negativity, then fact-check properties against reviews. For photos, generate several circle candidates per image and pick the one that is symmetric, large and relevant to the caption.

Questions this case answers
  • How to generate short and long texts for many objects with an LLM that are safe and factual by chaining narrow correction models around the LLM, each fixing one failure seen in production
  • How to choose a representative image crop for each object from its photos so that the crop matches the caption and is judged well by editors
  • How to generate short and long venue descriptions with an LLM that are safe to show in a map product
  • How to filter refusals, filler praise and invented negativity out of LLM review summaries

Tools and methods

  • YandexGPT — generates captions and descriptions from reviews, extracts properties, removes negativity and filler
  • Stupid-answer filter and style-fixing model — drop refusals and fragments, enforce editorial policy
  • Property extraction model (BERT) — properties from reviews with sentiment, used for fact-checking
  • PSPNet + EfficientNet, Gaussian Mixture, PCA — segmentation and candidates for photo circles
Continue reading — it's free

Google or a one-time email code. No password.