We are currently struggling to understand how the Shopify App Store rating system behaves in practice.
We understand that the displayed rating is not a simple arithmetic average and that Shopify uses a weighting model based on more recent and “high-confidence” signals. However, the actual outcome still feels difficult to reconcile with the visible review pattern.
A recent example from our side:
-
On Sunday, we received two new 5-star reviews.
-
Before Sunday, our app had already been showing an average of 4.8 in the App Store.
-
Today, we then received a single 1-star review caused by a misunderstanding that we are currently resolving directly with the merchant.
-
Immediately after that, the displayed average dropped to 4.7.
What makes this even harder to understand is that between an earlier 1-star review and today, we received a total of four 5-star reviews and no other ratings in between.
From our perspective, this raises a fair question:
How can multiple recent 5-star reviews have so little visible stabilizing effect, while a single 1-star review appears to move the displayed average so quickly?
We fully understand and support the idea behind weighted ratings if the goal is to reflect recent merchant experience more accurately. But in cases like this, the result feels disproportionate and not very transparent from a partner perspective.
We would really appreciate more clarity on questions like:
-
How strongly are recent negative reviews weighted compared to recent positive ones?
-
Are there thresholds, confidence multipliers, or other mechanisms that can make the displayed score react this sharply?
-
Is there any public documentation that helps partners understand why the visible rating may differ so much from the apparent review trend?
We are not questioning the need for a quality-focused rating system. We are simply asking for more transparency, because otherwise it becomes very difficult to understand or explain these rating movements internally and externally.
**EDIT: Another App in the same category looks like this:**
Based on the rating counts shown in the screenshot, the actual average is 4.62, which would normally round to 4.6, not 4.7. That is why we are struggling to understand how the displayed score changes so sharply in response to individual reviews.
3 Likes
Hey
App Store reviews are weighted, as you mentioned. From what I’ve been able to gather, the recency of a review seems to have the biggest impact on how strongly it’s weighted. I’d also expect there are some trust signals involved as well, such as how long the merchant actually used the app before leaving a review. Looking at your app, I can see most users had been using it for quite a while before reviewing, which is great. I’d also imagine signals like whether the store appears active are taken into account.
I doubt Shopify would share much detail about the exact scoring, as that information would be extremely useful for review farms, which are already a significant problem.
I can see the 1-star review is still the most recent on your app, so it will be interesting to see how quickly the overall score shifts back up once a few more positive reviews come in above it. I do sometimes wonder whether 1-star reviews are weighted more heavily than higher ratings.
Personally, I think the review layout on your app looks great. Having a few negative reviews is completely natural and can actually help build trust, as it shows the reviews are genuine rather than a wall of perfect 5-star ratings.
5 Likes
Hi,
thanks a lot for your thoughtful reply and for taking the time to share your perspective.
What you’re saying makes sense, especially regarding recency and potential trust signals such as store activity and how long a merchant has actually used the app before leaving a review. We also understand why Shopify would likely keep the exact weighting logic private.
What surprised us was simply how strong the immediate impact was. We had already built up a solid average before, received additional 5-star reviews, and yet one recent 1-star review caused a very noticeable drop right away. That felt disproportionate from our point of view, especially since the 1-star review appears to be based on a misunderstanding that we are currently clarifying with the merchant.
That said, we really appreciate your balanced view. You’re absolutely right that a completely perfect wall of reviews can sometimes look less authentic, and in general we also believe that a realistic mix of feedback can increase trust.
We’ll keep a close eye on what happens next — for example, if a new 5-star review comes in and how that affects the overall rating — and we’ll share an update here in the forum thread once we have a clearer picture.
Thanks again for your honest and constructive input — much appreciated.
Best,
Stefan
2 Likes
Basically a 1 star review by a mentally unstable user who never sold a product, offsets 100 5 star reviews from real store owners
7 Likes
That’s also your experience so far.
2 Likes
Geez, around less than 24 hours after I posted this, someone on Shopify Plus left a 1 star review with no text. After relentlessly trying to contact them, they finally replied after 3 days and said they left the review “by mistake”
1 Like
We just received another 5 Stars review but the average did not change back from 4.7 to 4.8
I have to correct my first reply: That’s also our experience so far.
Hey @Stefan_Neuser , interesting, it does seem like in this case the 1-star review is carrying more weight. I’ve taken another look at your listing, and I think I can see why.
The recent 1-star review came from someone who’s been using your app for over four years. That kind of long-term usage likely adds a lot of weight, as it suggests they know the app inside out. So I can see why Shopify would treat this as a very strong signal, and I think that makes sense.
That said, it’s frustrating that the review seems to stem from pricing confusion. It does feel like one you could potentially resolve by reaching out directly, especially given they’ve been with you for so long. If you’re able to turn that into a 5-star review, I’d imagine that same long-term weighting would work in your favour.
Thanks @Ollie_autoBlogger again for those insights. We’re currently still waiting for a response from the merchant, but at the moment it looks like there’s a good chance we can resolve the situation and either improve the review or have it removed, since this really seems to have come down to a misunderstanding. It will definitely be interesting to see how the overall rating changes afterwards.
That said, your point about Shopify using a more sophisticated algorithm to prevent review farming is exactly what I keep wondering about as well. For apps with a relatively small number of reviews, I’m not entirely sure how much sense such a complex weighting model really makes. In our case, for example, the app has collected only 336 reviews over more than six years. With numbers like that, it feels highly unlikely that there is any meaningful risk of large-scale review fraud or purchased reviews.
That’s why I do wonder whether a simpler and more transparent weighting system would perhaps make more sense for smaller apps, especially since a single heavily weighted review can have a disproportionate impact on the overall rating. From a developer perspective, that can feel quite harsh, particularly when the issue behind the review is based on a misunderstanding rather than the actual quality of the app itself.
PS: @Liam-Shopify - Can it be that @Ollie_autoBlogger is a bot?
Hey,
I do think it makes sense for Shopify to place significant weight on reviews from stores that have used an app over a long period of time, and I don’t personally think that should change.
Good luck getting the review updated, that’s probably your best bet here.
P.S. You got me, I’m definitely a bot, the kind that somehow built 6 apps, has been on the App Store for 3 years, and is oddly passionate about calling out fake reviews.
@Ollie_autoBlogger How did you figure out about which apps we are talking about? I never mentioned it so far. But it seems that you found it out and took a deeper look. How?
Nothing fancy, just a quick Google. I spend a fair bit of time looking at app listings and reviews, so I’m pretty good at navigating the space.
Hey @Ollie_autoBlogger
just to clarify where that assumption came from — your name includes “Autoblogger,” and because there are projects out there where bots automatically generate or publish blog content, that was simply my first association.
So it really wasn’t meant as a serious accusation.
Just out of curiosity, which Shopify apps are you running?
Stefan
No worries at all! Yeah, autoBlogger is my main app, hence why it’s in my username. It focuses on properly structured, SEO-driven content rather than the usual spammy stuff. I’ve got a few others as well. You’re welcome to check them out: My Partner Profile
@Ollie_autoBlogger These are a bunch of really well made apps, Congrats 
Was it hard to get the built for Shopify status?
1 Like
Thanks so much! Yours look great too. And no worries at all for snooping, I noticed that review got sorted as well, nice one.
Regarding Built for Shopify status, I’ve only got it on two of my apps so far. I’ve found that as long as everything is built using Polaris, the trickiest part is actually hitting that 50 net installs, getting 5+ reviews, and enough activity on the benchmark criteria.
Is there any particular reason you haven’t gone for it on your apps? I’d definitely say it’s worth the effort - It gives you a search ranking boost and trust signal if nothing else. Happy to share anything I’ve learned if it helps. I don’t think we can DM on the dev community, but feel free to email me anytime at support@autoblogger.bot 
Why not having a video call sharing some experience?
@Ollie_autoBlogger ? So you are a bot?
Hey, sorry missed your last message. Your welcome to email me if you needed any advice, but na, I don’t generally do video calls.
