Thank you everyone for the updates and for sharing the specific business impacts you’re experiencing.
I am continuing to ensure your feedback is heard by the appropriate teams. Given the severity of the issues some of you have mentioned, I’d recommend reaching out to our support teams directly as well. This will allow us to dig more deeply into your specific store configurations and get a detailed picture of the exact blockers you’re experiencing.
When you do reach out, please reference this community thread and include specific details about your testing setup, errors you are experiencing, and any workarounds you’ve tried. This context will help support investigate more effectively.
Feel free to continue sharing updates and resolutions you’ve found here as well. This thread is a valuable resource for other developers running in to similar limitations.
This is a total pain, we cannot even crawl our own site to verify 404 or 301 issues we would like to address due to this issue. At a min we should be able to whitelist something so we can crawl our own site!
Not really because its made for merchants, not partners. Partners need to scan many different merchant stores, and asking each merchant to generate a 3 month signature that we need to manage for each store is a bit wild. Feels more like a solution for helping merchants use big SEO SCANNING sites like screaming frog, not for us. Am I wrong here?
It appears that Shopify has been battling these giant scanning sites, and we got caught up into it. As actual shopify partners, we should have a more convenient solution. Whitelist our servers for any merchant domains that have our apps installed… Or give us a signature that works on all our merchant store domains.
as @jason_engage mentioned a solution that validates a partner rather than a store would be much better. Especially that you could easily track what partners are doing with their keys and to what degree.
While we could do a rotation of that sig every 3 months - that’s not a big deal I can’t imagine how thousands of our customers will be doing that every 3 months as an additional task just to keep running our app. Not to mention that they would need to do that for every store separately, and pass it to us, that is A LOT of additional work.
Maybe making that signature generation available via API would make things a bit easier but this still adds complexity.
Also, while I’m assuming this will bypass the CF protection, I’m guessing there still is some rate limitation? Could we get some documentation also on that?
This is just something that allows SEO / crawlers to scrape data from storefronts. We are trying to run automation tests on our staging environment storefront so we can do regression testing before we do releases for our production store.
We need something that will allow us to access the checkout endpoints reasonably without getting hit by 429 errors from cloudflare and shopify.
We need a way to get whitelisted so we can reasonably run checkouts.
I wrote a couple of cypress tests that “crawl” a sample dev store that I have, but I can’t hit the limits to reliably get a 429 error. I would like to be able to write a couple of tests that will verify that providing the signatures on my request headers, actually makes a difference.
Is there any way for me to verify that the signatures are working? I was thinking something like some information on the responses, but even any information on what the current limits are would be really helpful!
I’m encountering persistent HTTP 429 errors when crawling a Shopify site – even at relatively conservative speeds. A few months ago, I could crawl the full site within an hour without issue. Now, unless I slow the crawl rate to an extremely low level, the server starts throwing 429s almost immediately.
While I understand the need for rate limiting, it’s becoming impractical to gather data or run audits when a single crawl has to span several hours just to avoid being blocked. Is there any chance this might be adjusted in the future?