Problem
A team has a large stream of user texts, from one app's reviews to many millions of messages, and needs to know what users think of a specific company, product or feature. One sentiment score per text does not help: in long texts it averages praise and complaints into a "mean temperature across the hospital". The object is hard to identify, and positive and negative examples are rare: the speaker's data had about 200-300 per 5,000 neutral ones.
Example from the article: measuring how much users like a company (the speaker's illustration is "Roga i Kopyta", also "Rogokop" after a rebrand), where the verdict must concern that company and not the whole comment.
Solution in brief
Judge sentiment for a pair "object or aspect + text". Cut everything unrelated to the object from long texts, then put a simple model on embeddings of what is left. Fight imbalance with candidate pre-selection for labeling, augmentation and a split stratified by label and object. Local-context models (LCF, LSA) or an instruction-tuned model are the heavier options.
Questions this case answers
- How to measure sentiment toward one specific object inside long texts that mention many objects
- How to build a training set for targeted sentiment when the positive and negative examples are rare and find annotation errors in existing labels
- How to measure what users praise or criticize about a specific brand, product or feature in long user-generated texts
- How to build a training set for aspect-level sentiment when positive and negative examples are rare
Tools and methods
- Object and aspect sentiment — label per (object or aspect, text) pair, three classes.
- Filtered-text baseline — embeddings, then logistic regression, LightGBM or CatBoost.
- PyABSA, LCF, LSA — ready ABSA models with local context.
- Tk-Instruct — instruction-based model, SOTA on open datasets at the time.
- Toloka — crowd annotation.
Google or a one-time email code. No password.