From Platonic Dialogues to Strong AI, or a Few Words on Multimodal RAG for LLMs

Roman Derunets25 May 2024

Problem

A team that wants an LLM to answer questions about images and sound usually trains projection layers between a visual encoder and the LLM, or trains an end-to-end multimodal model. Both routes need a lot of multimodal data, compute and time, and projection layers keep the link between modalities weak. An LLM that orchestrates expert models has no explicit connections between modalities. A small research team from Novosibirsk State University wanted a multimodal assistant without such training.

Example from the article: the first version, given a photo of pears and apples, returned an answer about the Army of Christ and religious organizations.

Solution in brief

The system turns an image or sound into one trimodal embedding, retrieves the closest text paragraphs from a vectorized knowledge base (the paragraphs the model "recalls"), and passes them to an unchanged LLM. Small unimodal models describe the input in text. That description reranks the paragraphs and goes into the prompt.

Questions this case answers
  • How to answer questions about non-text inputs with a frozen language model by retrieving text passages through a shared embedding space, without training projection layers
  • How to rerank retrieved passages using short descriptions of the input produced by small unimodal models
  • How to build a multimodal LLM that answers questions about images and audio without training projection layers
  • How to retrieve text knowledge-base paragraphs for an image or sound query using a shared embedding space

Tools and methods

  • ONE-PEACE-4B — shared embedder for text, image and sound
  • PCA — 1536 to 300 dimensions
  • Annoy — nearest-neighbor index over Wikipedia paragraphs
  • Mistral-7B-Instruct (v0.1, v0.2), Phi-3-Mini-3.8B — answer generators
  • BLIP, Audio Spectrogram Transformer (AST), Whisper-Medium — caption, sound class, speech transcript
  • MPNet-Base — text encoder for reranking
  • BERTScore, Tiny LVLM Evaluation — offline evaluation
Continue reading — it's free

Google or a one-time email code. No password.