Problem
Brand teams learn what customers think from surveys, which have sampling bias and cover few people. Social media holds far more opinions, but a month of posts about one company is small (about 20 million words, against about 100 billion for the Google News Word2Vec model) and noisy. Embeddings trained on such data change from run to run, so a month-to-month score moves for reasons unrelated to the brand. There is also no ground truth for "brand perception" to pick the best method against.
Example from the article: Airbnb tracks perceptions such as "belonging", "connected" and "reliable" from posts on 19 platforms, and checks how one brand theme moves after a campaign.
Solution in brief
Train word embeddings on each month of brand-related posts and score a concept by its cosine similarity to the brand word. To make the series stable, average the scores of 30 models trained with the same settings, and convert the similarity into a rank-based score (). Choose between methods with a variance metric across periods.
Questions this case answers
- How to track the strength of association between an entity and a set of concepts over time from a small noisy text corpus when no ground truth exists
- How to make a score derived from stochastically trained embeddings comparable across time windows
- How to measure how strongly a brand is associated with concepts such as 'home' or 'reliable' from social media posts?
- How to get a stable month-over-month time series from word embeddings trained on a small, noisy corpus?
Tools and methods
- Word2Vec (CBOW, Gensim) — baseline in-domain embeddings.
- FastText with sentiment tags — subword vectors, words tagged POSITIVE, NEGATIVE or NEUTRAL.
- DeBERTa trained from scratch (Transformers) — contextual embeddings; the best fit for this data.
- Cosine similarity to the brand word — the perception score.
- Averaging over repeated trainings — reduces run-to-run noise.
- Rank-based score — keeps relative order, not absolute distances.
Google or a one-time email code. No password.