From print to digital: Making weekly flyers shoppable at Instacart through computer vision and LLMs

Prithvi Srinivasan9 Feb 2026

Problem

Instacart wants every deal in a retailer's weekly flyer to be clickable and linked to catalog products. By hand this meant drawing a box around each deal and matching it to products, 3–4 hours per flyer. With dozens of retailers uploading weekly, that reached hundreds of hours a week and forced retailers to send flyers well before the deals went live. Layouts range from simple grids to dense spreads.

Example from the article: SAM marked each coffee bean and each scoop of ice cream as its own segment, so the boxes could not be merged into deals.

Solution in brief

A two-phase pipeline. First, a custom layer on Meta's Segment Anything Model (SAM) cleans its output into one box per deal: remove text-only boxes, fuse overlapping boxes, optionally add contour detection, filter noise. Second, OCR text plus the image go to an LLM that writes search queries, and the existing ANN search and an LLM ranker pick the catalog products.

Questions this case answers
  • How to turn a dense image of many items into clickable regions linked to catalog records when general segmentation models over-segment and multimodal LLMs give imprecise coordinates
  • How to match items shown in an image to a text-indexed catalog by converting the image into search queries and reusing the existing retrieval stack
  • How to find the clickable product regions in a promotional flyer image without drawing boxes by hand?
  • How to match each deal in a printed promo image to the right items in a product catalog?

Tools and methods

  • Segment Anything Model (SAM) — base segmenter that proposes candidate regions
  • Weighted Boxes Fusion (WBF) — merges overlapping boxes
  • Contour detection — second box source for dense flyers
  • Heuristic and ML box filters — drop false positives
  • PaddleOCR — reads text in each box
  • LLM — splits a box into queries and attributes, then ranks candidates
  • Search ANN index and feature store — retrieve and describe candidate products
Continue reading — it's free

Google or a one-time email code. No password.