Beyond global sentiment: object, aspect sentiments

Maxim Panteleev27 May 2023

Problem

A team has a large stream of user texts, from one app's reviews to many millions of messages, and needs to know what users think of a specific company, product or feature. One sentiment score per text does not help: in long texts it averages praise and complaints into a "mean temperature across the hospital". The object is hard to identify, and positive and negative examples are rare: the speaker's data had about 200-300 per 5,000 neutral ones.

Example from the article: measuring how much users like a company (the speaker's illustration is "Roga i Kopyta", also "Rogokop" after a rebrand), where the verdict must concern that company and not the whole comment.

Solution in brief

Judge sentiment for a pair "object or aspect + text". Cut everything unrelated to the object from long texts, then put a simple model on embeddings of what is left. Fight imbalance with candidate pre-selection for labeling, augmentation and a split stratified by label and object. Local-context models (LCF, LSA) or an instruction-tuned model are the heavier options.

Questions this case answers
  • How to measure sentiment toward one specific object inside long texts that mention many objects
  • How to build a training set for targeted sentiment when the positive and negative examples are rare and find annotation errors in existing labels
  • How to measure what users praise or criticize about a specific brand, product or feature in long user-generated texts
  • How to build a training set for aspect-level sentiment when positive and negative examples are rare

Tools and methods

  • Object and aspect sentiment — label per (object or aspect, text) pair, three classes.
  • Filtered-text baseline — embeddings, then logistic regression, LightGBM or CatBoost.
  • PyABSA, LCF, LSA — ready ABSA models with local context.
  • Tk-Instruct — instruction-based model, SOTA on open datasets at the time.
  • Toloka — crowd annotation.
Continue reading — it's free

Google or a one-time email code. No password.