Scaling Catalog Attribute Extraction with Multi-modal LLMs

Shih-Ting Lin1 Aug 2025

Problem

Instacart's search filters, size and flavor pickers and badges such as "Gluten-Free" depend on structured attributes across millions of SKUs and thousands of categories. SQL rules work only for simple cases like an "organic" keyword. Per-attribute text models need their own labeled data, training and maintenance. Both approaches read only product text, so they miss values printed only on the package image.

Example from the article: a product description lists the flavors of all variants (Orange, Grape, Strawberry), and the old system could not tell that the primary flavor of this item is Orange.

Solution in brief

PARSE is a self-serve platform where a team defines an attribute, a prompt and an input SQL, then runs a multi-modal LLM over the catalog. A second "yes/no" prompt gives a confidence score, and low-confidence values go to human auditors. Samples are checked by humans or an LLM judge in development and in production.

Questions this case answers
  • How to extract many different structured fields from unstructured text and images at scale when training a separate model per field is too slow and expensive
  • How to attach a confidence score to a generative model's extracted value so that only doubtful values go to human review
  • How to extract structured product attributes from titles, descriptions and images for millions of SKUs without training a model per attribute?
  • How to get a usable confidence score for an LLM-extracted attribute value and use it to route values to human review?

Tools and methods

  • Multi-modal LLM extraction — reads text and images, zero-shot or with few-shot examples
  • Self-verification confidence score — a second prompt checks the extracted value; probability of "yes" is the score
  • LLM cascade — several extraction algorithms trade cost against accuracy per attribute
  • Human-in-the-loop quality screening — auditors label samples and review low-confidence values
  • LLM-as-a-judge — speeds up evaluation of extraction results
Continue reading — it's free

Google or a one-time email code. No password.