Problem
Product and QA teams of a large app get a stream of store reviews from many devices and OS versions. Testing cannot cover that variety, and logs show crashes but not what users find confusing or valuable. During Mercari's 2022 beta of a rewritten app, the QA and Voice of Customer (VoC) teams sorted feedback by hand. They struggled with duplicates, language barriers, routing items to the right team, and daily volume.
Example from the article: after the beta ended, the QA engineer who ran the manual analysis proposed automating categorization to process more feedback. A classifier built at a hackathon later fed weekly summaries of Google Play and App Store reviews.
Solution in brief
Pick a small set of feedback categories, hand-label about 100 reviews per category, and train a classifier. Run it on every new review, and post a weekly Slack digest of the categories that matter (bugs, improvement requests, hot topics). Add a filterable dashboard.
Questions this case answers
- How to turn a high-volume stream of free-text user feedback into a recurring, prioritized list of issues and requests when no labeled data exists
- How to give non-technical stakeholders self-service access to classified text feedback without duplicating the data across teams
- How to turn thousands of app-store reviews into a weekly list of bugs and improvement requests for product teams?
- How to prioritize investigations and tickets by how many users mention a topic?
Tools and methods
- Review classifier — assigns each review to one of eight categories; built in a 3-day hackathon.
- Hand labeling — about 100 reviews per category to train and show the model works.
- Weekly Slack report — hot topics, bugs, improvement requests, other.
- Interactive dashboard — filters by date, rating, category, feature and tags; keyword search in English and Japanese.
Google or a one-time email code. No password.
