Bug: 429 page errors are too aggressive

Thank you everyone for the updates and for sharing the specific business impacts you’re experiencing.

I am continuing to ensure your feedback is heard by the appropriate teams. Given the severity of the issues some of you have mentioned, I’d recommend reaching out to our support teams directly as well. This will allow us to dig more deeply into your specific store configurations and get a detailed picture of the exact blockers you’re experiencing.

When you do reach out, please reference this community thread and include specific details about your testing setup, errors you are experiencing, and any workarounds you’ve tried. This context will help support investigate more effectively.

Feel free to continue sharing updates and resolutions you’ve found here as well. This thread is a valuable resource for other developers running in to similar limitations.

How do we reach out to the specific support teams?

@jason_engage you can connect with support teams via the “Support” tab in your partner dash, and then selecting “contact Partner Support.”

C’mon, thats like entry level support. It’ll be an intern or new hire and we’ll be back at square 1. Hah

This is a total pain, we cannot even crawl our own site to verify 404 or 301 issues we would like to address due to this issue. At a min we should be able to whitelist something so we can crawl our own site!

@jason_engage @Joshua_Fricke @TPA_Admin @Maurice_Foreman is this change gonna help?

Not really because its made for merchants, not partners. Partners need to scan many different merchant stores, and asking each merchant to generate a 3 month signature that we need to manage for each store is a bit wild. Feels more like a solution for helping merchants use big SEO SCANNING sites like screaming frog, not for us. Am I wrong here?

It appears that Shopify has been battling these giant scanning sites, and we got caught up into it. As actual shopify partners, we should have a more convenient solution. Whitelist our servers for any merchant domains that have our apps installed… Or give us a signature that works on all our merchant store domains.

as @jason_engage mentioned a solution that validates a partner rather than a store would be much better. Especially that you could easily track what partners are doing with their keys and to what degree.

While we could do a rotation of that sig every 3 months - that’s not a big deal I can’t imagine how thousands of our customers will be doing that every 3 months as an additional task just to keep running our app. Not to mention that they would need to do that for every store separately, and pass it to us, that is A LOT of additional work.

Maybe making that signature generation available via API would make things a bit easier but this still adds complexity.

Also, while I’m assuming this will bypass the CF protection, I’m guessing there still is some rate limitation? Could we get some documentation also on that?

This is just something that allows SEO / crawlers to scrape data from storefronts. We are trying to run automation tests on our staging environment storefront so we can do regression testing before we do releases for our production store.

We need something that will allow us to access the checkout endpoints reasonably without getting hit by 429 errors from cloudflare and shopify.

We need a way to get whitelisted so we can reasonably run checkouts.

Hi all,

After going through the documentation here:

https://help.shopify.com/en/manual/promoting-marketing/seo/crawling-your-store

I wrote a couple of cypress tests that “crawl” a sample dev store that I have, but I can’t hit the limits to reliably get a 429 error. I would like to be able to write a couple of tests that will verify that providing the signatures on my request headers, actually makes a difference.

Is there any way for me to verify that the signatures are working? I was thinking something like some information on the responses, but even any information on what the current limits are would be really helpful!

Hi everyone,

I’m encountering persistent HTTP 429 errors when crawling a Shopify site – even at relatively conservative speeds. A few months ago, I could crawl the full site within an hour without issue. Now, unless I slow the crawl rate to an extremely low level, the server starts throwing 429s almost immediately.

While I understand the need for rate limiting, it’s becoming impractical to gather data or run audits when a single crawl has to span several hours just to avoid being blocked. Is there any chance this might be adjusted in the future?

GET 429 errors while developing with the CLI on a store page. This has blocked me from working on a store today. Why would this be a limit while using the CLI ?

Not sure if anything changed about this recently, but we started seeing this on one of our repo’s single E2E test earlier today; as far as I know it’s never impacted us before, and we haven’t changed our E2E test for months/years. Now webkit and Safari always get a 429 just trying to load the store’s index page, making the test consistently fail and blocking our ability to deploy new versions.

I asked Shopify why my store was blocking customers. They told me it was working correctly.

For weeks support told me nothing was wrong and the platform was performing as expected, while Shopify staff were posting in this very thread acknowledging the VPN problem is real and unresolved. I sent them this post. It made no difference.

They would only accept a HAR file showing the full error. Not the videos, not the screenshots, not any of the evidence I and my customers kept sending showing something was definitely wrong. And it’s a tricky error to catch, because it’s intermittent and by the time you see it, it’s gone.

What happened

On 29 July I finally captured it. I opened my own shop in Chrome. Not a script. Not a crawler. Me, clicking through to my homepage like any customer would.

I got a blank white page with two words on it: local_rate_limited. No branding. No products. No way through. My entire store, gone.

I sent the request ID to support. Shopify pulled the logs at edge and origin. A human being looked at a session where a real person was refused entry to a real shop, and wrote back:

“Shopify’s bot protection assessed this session as automated traffic and applied rate limiting accordingly. This is expected behavior for that type of connection… There is no platform error.”

Why I was flagged as a bot

A VPN was running. Specifically McAfee Total Protection, ordinary consumer antivirus. It switches its VPN on automatically on unsecured Wi-Fi, and McAfee advertises it for shopping and banking. Norton does it. Google One does it. Apple’s Private Relay does it.

That’s not a datacentre. That’s a shopper with antivirus software.

Shopify is openly admitting it blocks VPN traffic, while behaving as though only a handful of unusual people use one.

And if that’s the response, what was the point of asking us for request IDs?

It isn’t just me — and it costs you twice

I’d already sent Shopify video of an actual customer, not me, trying to reach my store from a Facebook link. Three devices. Same blank page every time.

Then they gave me my numbers. 1,688 of these served to my storefront in five days. 91% attributed to “VPNs, hosting providers, datacenters, or Meta infrastructure IPs—not from residential customer connections.” Edge logs only go back five days, so whatever happened before that is gone.

Read that list again. Meta infrastructure. Some Meta traffic genuinely is crawlers and does need limiting. But traffic from Facebook and Instagram also arrives through Meta’s in-app browser, which is how a huge number of paid-social customers reach your store.

And here’s the kicker. You pay Meta to deliver that customer. Meta charges you for the click. Then Shopify refuses to serve them the page. So you have paid for a customer you were never going to be allowed to sell to, and you lose the sale on top. You are being charged twice for the same block, and nothing in your reporting tells you it happened.

I spent £3,531.72 with Meta last month. My Facebook conversion rate has gone from 3.33% in May to 0.78% now.

It isn’t confined to my store

I reproduced the same block on other people’s stores. Shopify wouldn’t accept those because they weren’t my account. I offered to get written permission from the owners and was told they’d have to contact support themselves. One did.

They were told it would be logged as a “frustration.”

So, publicly and on the record:

Shopify is treating VPNs as something a minority of customers use, when it is now standard in most antivirus software. When is Shopify going to stop turning away real customers, and admit the platform is currently broken?

i’m getting the same as you and it started pretty much exactly the same time. I have a post that’s 25 days old too saying the same issue. i’ve just posted additional info in this thread. there is an issue but they are ignoring it.

That sounds very frustrating @Hannah_Hoskins :flushed_face: At least as an app developer this only affected my automated testing, but costing real stores customers and sales is not cool :disappointed_face: Unfortunately it can be hard to get the right Shopify peoples’ attention when there’s a technical issue, which is definitely a “frustration”; unfortunately I can’t offer any further help or advice for your scenario because I was able to auth my test runner as an “allowed” crawler, which of course isn’t relevant in the context of a real store’s customers being blocked. I hope you’re able to get somewhere with this soon! :crossed_fingers:

Apparently I was assigned a ‘specialist’ and a special team were looking into it but their response was the ‘VPNs are unusual’ and when I called that out they admitted in writing they are loosing me sales and have no intention of doing a fix…