Problem
LINE OpenChat has a main screen built around chatrooms. Users who already found a good room rarely come back to browse more rooms, yet main-screen visits are a core KPI tied to ad revenue. The team wanted to show message content instead, grouped by topic keywords. Raw message frequency does not work here. Greetings, thanks and laughter dominate, and copy-pasted promotions repeat thousands of times. Single messages are also too short and messy to show on their own.
Example from the article: the keyword 梅雨入り (start of the rainy season) should be flagged as trending in the week the season began in Japan, while 今週 (this week) should not.
Solution in brief
A word is trending when its share of messages today is statistically higher than a week ago, measured by a two-proportion Z statistic. Before counting, near-duplicate messages are collapsed with MinHash. After scoring, an NPMI filter built from a few seed words removes unwanted topics, and an MMR-style re-ranking keeps top keywords from overlapping.
Questions this case answers
- How to surface topics whose usage is rising in a high-volume text stream dominated by constant everyday phrases and bulk-duplicated content
- How to exclude a sensitive topic from an automatically generated keyword list when its vocabulary cannot be enumerated in advance
- How to find keywords that are trending in a stream of chat messages instead of everyday words like greetings?
- How to stop copy-pasted promotional messages from being picked up as trending topics?
Tools and methods
- Two-proportion Z-test — scores the daily rise in word frequency against a baseline day.
- MinHash clustering — removes copy-pasted and near-identical messages before counting.
- NPMI (normalized pointwise mutual information) — measures word co-occurrence to expand a seed list of unwanted words.
- MMR (maximal marginal relevance) — re-ranks keywords with a penalty for similarity to those already chosen.
Google or a one-time email code. No password.
