An iceberg in a calm grey sea, its small tip above the water and its far larger body glowing aqua below. One person in a rowing boat looks at the tip.

Feedback analysis

Review sentiment analysis with AI: what star ratings miss

Review sentiment analysis with AI: why a five-star review can still report a bug, what one score per review hides, and how to check it.

Nirupa Barad

5 min read

Share

Your star rating is high, so you assume customers are happy. Then a five-star review turns up that says the app logs people out every day. Review sentiment analysis sorts reviews into happy, unhappy or in between. Done well, it reads the words, not only the stars, so that complaint still gets seen.

A bug that sits inside five-star reviews stays unfixed until the rating drops.

Key takeaways

  • Stars and words are two separate things. A happy customer can give five stars and still report a problem in the text.
  • One label or one score per review hides mixed reviews. Praise and a complaint can cancel out and look like nothing.
  • Before you trust a sentiment score, find a good review with a complaint in it and see what the tool says about it.
  • Sarcasm can be misread, and a one-word review holds no reason. So read the customers' own words before you act.

What is review sentiment analysis?

It means reading each review and deciding how the writer feels. Sentiment is that feeling: how happy or unhappy they are.

Some tools give each review one label: positive, negative or neutral. Others give a number, where higher means happier.

Today an AI tool can do this reading for you. Then it adds the reviews up into one sentiment score, such as "mostly positive this month."

That score is quick to read. It is also easy to trust too much.

What do star ratings miss?

Stars and words are two separate things. Apple's help page says people can rate an app from 1 to 5 stars, and can also write a review (App Store Connect Help). Nothing makes the two agree.

Researchers who read Android app reviews by hand found ratings that did not fit the text. Their example was a review that said "Awesome App" and gave one star (Aralikatte and others, 2017).

The opposite case costs you more. A customer who likes your app gives five stars, then mentions the bug in the second sentence. Our step-by-step guide to review analysis has a real case: some of the reviews carrying a team's top complaint were four stars.

What does one sentiment score miss?

It misses the review that says two things. Take this one: "Love the new design. But it logs me out every day."

A tool that gives one label has to pick. If it says positive, the bug drops out of sight. If the praise and the complaint cancel out, the review looks neutral, like one with nothing to say.

Google's documentation for its own text analysis service describes this. A neutral score can mean little feeling, or it can mean "mixed emotions" that cancel each other out (Google Cloud documentation).

Here are four reviews, made up for this post and not from any customer.

What the customer wrote Stars One label says What it tells you
"Love the new design. But it logs me out every day." Five Positive A login problem, from a happy customer
"Great, another update that broke search." Three Positive, because of "great" Search broke after the app update
"Used to be my favorite app. The ads ruined it." Two Negative The reason is ads
"Fine." Three Neutral Nothing to act on

The label column tells you a mood. Only the last column tells you what to fix.

How do you check review sentiment by hand?

You read the words first and the stars second.

  1. Cover the star column and read the review.
  2. Split it in two: what they like, and what they complain about.
  3. Write the complaint in a few words, such as "logged out every day."
  4. Put a mark next to that complaint each time it comes up again.

This works for a few dozen reviews. Past that it is slow, and the star average is easier to read.

What should you check before you trust a sentiment score?

Run three checks on whatever AI tool gives you the score.

  1. Find a good review with a "but" in it. See what the tool says about that review. If it says "positive" and nothing more, the score is hiding complaints.
  2. Ask what the unhappy part is about. Try: "What do customers who like the app still complain about?" A score with no reasons is a mood, not a list of things to fix.
  3. Open the words. Every label should lead you to the sentence behind it. If you cannot read the sentence, you cannot check the label.

The guide on how to analyze customer feedback with AI covers checking in more depth.

How Metricsense does this

Each step starts with the problem, then says what Metricsense does about it.

  1. Your reviews sit in the app stores. Metricsense connects to the App Store and Google Play. Anything else comes in as a CSV, a plain spreadsheet file.
  2. A score tells you the mood, not the reason. You describe what to find in plain words, such as "What do customers who like the app still complain about?" Metricsense calls a question like this an AI Insight. You do not sort anything into groups by hand, and you do not teach the AI with examples first.
  3. You cannot tell if three complaints are one problem. Metricsense groups reviews by reason, most common first. It calls these groups themes. Sentiment analysis is included too, so you still see how happy or unhappy reviewers are.
  4. A label you cannot check is a guess. Every finding links to the exact customer quote behind it, so you can read it before you act.
  5. You want to watch a complaint from one month to the next. Trend tracking shows each group over time.

Here is what comes back for the question in step 2. Illustrative data, not actual customer data.

Logged out every day (most common)

  • "Love the new design. But it logs me out every day."
  • "Best budgeting app I have tried. Please fix the login, I sign in every morning."
  • "Great features. I just wish it remembered who I am."

Search broke after the app update

  • "Really useful app. Search has not worked since the update."

What you learn from this: the top complaint comes from people who say they love the app. A star average would count every one of them as happy.

What stays with a person?

Deciding what to fix first stays with you. A complaint from happy customers may matter more than a louder one from angry customers, or less. Your team knows which.

The reply stays with a person too. Someone on your team should answer the customer who wrote in.

What does review sentiment analysis get wrong?

Sarcasm, and reviews too short to hold a reason. "Great, another update that broke search" can be read as praise.

A 2025 study of app reviews says older methods struggle with sarcasm (Zuhir and others, 2025). The same study found newer AI did better. Still, do not expect any tool to get every review right.

A review that says only "Fine." has no reason in it. There is nothing there for a tool to find. That is why the quotes matter more than the score.

Try it on your best reviews

Connect your app store or upload a CSV. Then ask what your happy customers still complain about. Start free with Metricsense.

Share this article

Back to the blog