ESQA: Event Sequences analysis through Question Answering

Irina Abdullaeva25 May 2024

Problem

A bank has many tasks on the same customer histories: next purchase, credit default, age, gender. The usual setup builds a separate model for each one (boosting, a neural network or a statistical model), trained only for that task. Every new question means a new model, new features and a new training run.

Example from the article: on AlfaBattle 2.0 one model answers questions about every attribute of the customer's next transaction (MCC code, amount, city, time to the next operation) as text.

Solution in brief

Turn every task on an event sequence into a question with a text answer. A trained encoder turns the history into embeddings, a Q-Former compresses them into a fixed number of vectors, and a frozen LLM reads them together with the question. One model then handles many tasks, and one pretrained only on questions about the existing history can still answer prediction questions.

Questions this case answers
  • How to answer many different predictive and descriptive questions about one entity's event sequence with a single model by casting each task as a text question for a frozen language model
  • How to compress a variable-length event history into a fixed number of vectors that a language model can read
  • How to predict a customer's next transactions and their attributes from bank transaction history with one model instead of one model per task
  • How to feed event sequences (transactions, clicks) into a frozen LLM

Tools and methods

  • ESQA (Event Sequence Question Answering) — the framing: task as question, solution as the text answer
  • Event sequence encoder — feature encoder, linear connector and a Whisper-tiny body (4 layers, 27M parameters)
  • Q-Former from BLIP-2 — compresses a long sequence into trainable queries for the LLM
  • Frozen LLM — reads the queries, the prompt and the question; its weights never change
  • CoLES, CPC, Barlow Twins, TabFormer, GPT, handcrafted features — baselines in the talk
Continue reading — it's free

Google or a one-time email code. No password.