SOTA Entity Recognition via LLM Distillation

Sergey Bogdanov28 Oct 2023

Problem

A team that needs entity recognition in several languages has two weak options. A BERT with a linear layer is specialized neither for the domain nor for the task, and most teams still start from the same 2018-era encoder (multilingual BERT, mBERT). Public NER datasets are limited: NER-BERT is large but English-only, MultiNERD is multilingual but has only 15 classes. Quality on non-English text stays low.

Example from the article: on eight languages (EN, DE, FR, ES, PT, PL, RU, NL) frozen mBERT reaches F1-macro 0.5206 in the single-language test.

Solution in brief

Label raw web text (OSCAR) with an LLM that invents entity types and describes each entity in free text. Train an encoder to match each token to the embedding of "type + description" instead of a fixed linear layer, freezing the lower six layers. The result is an open multilingual NER foundation model.

Questions this case answers
  • How to train a multilingual token classifier for open-ended entity types when no labeled data exists, by having a large language model label raw web text
  • How to make a frozen pretrained encoder match open-ended entity labels using intermediate layers and textual descriptions
  • How to build a multilingual named entity recognition model without a human-labeled dataset
  • How to use an LLM to label raw web text for entity recognition with open-ended classes

Tools and methods

  • LLM labeling (data-level distillation) — entity, concept and description for every sentence of raw text
  • Token-to-description similarity training — dot product, sigmoid, binary target matrix
  • Intermediate-layer features and partial freezing — layers 9 (or 6) and 12; lower six layers frozen
  • Logistic regression probe — tests the frozen model on each NER dataset
  • Data: OSCAR for training; CoNLL 2003, MIT Movie, MIT Restaurant, OntoNotes 5, BioNLP 2004, MultiNERD for evaluation
Continue reading — it's free

Google or a one-time email code. No password.