Developing a multi-label classification model for OpenChat hashtag prediction

Heewoong Park7 May 2025

Problem

LINE's OpenChat lets users create chat rooms with a name and an optional description. Hashtags make rooms easier to find, but users type them inside the free-text description and often skip them. The team wants to suggest hashtags on the creation screen (up to 30 are considered). The same model is later reused to tag rooms offline for ad matching, where a wrong tag leads to a wrong ad.

Example from the article: for the room "People who like Splatoon 2", four of the top five predicted tags began with "スプラ", so the list offered one idea five times.

Solution in brief

Treat tag suggestion as multi-label classification over a fixed, pre-reviewed tag set. Train a multilingual encoder with softmax cross-entropy against a label vector normalized to sum to 1, which fits a "show the top K" use case. Then add chosen tags to training inputs, remove near-duplicates with Maximal Marginal Relevance (MMR), and use two score thresholds for offline tagging.

Questions this case answers
  • How to suggest several relevant labels for a user-created text object when only the top K labels are shown and similar labels crowd the list
  • How to choose confidence thresholds for an offline multi-label tagger so that precision stays high while coverage stays acceptable
  • How to suggest tags for a new user-created group from its name and description when users type several tags themselves?
  • How to train a multi-label classifier when the product shows only the top K suggestions?

Tools and methods

  • Hugging Face Transformers, PyTorch — fine-tuning with a custom Trainer loss
  • multilingual-E5-large — pretrained encoder, best of six tested
  • scikit-learn — sparse label matrix, ndcg_score
  • Softmax cross-entropy, normalized multi-hot target — loss for top-K suggestion
  • Maximal Marginal Relevance (MMR) — removes near-duplicate tags
  • min_top1 and min_score thresholds — reliable output in batch tagging
Continue reading — it's free

Google or a one-time email code. No password.