Screaming still returns 429 even though the signature and HTTP header are configured correctly

[Wiempire_LLC]

In recent months, every time I try to crawl my Shopify website with Screaming Frog, Screaming Frog returns 429 response codes (Too many requests).

I configured the signature as instructed by Screaming Frog, and at first the problem was solved, but in the last two weeks it has not been letting me crawl anything, which is preventing me from working on my project properly.

How can I avoid the 429 errors from SF if I have already tried the signature and lowering the threads?

Thank you!

[Kellan-Shopify]

Hi @Wiempire_LLC,

We have some great documentation in our Help Center on how to setup signatures in your Shopify admin to enable web crawlers on your storefront without error.

• Shopify Help Center | Crawling your store

It does also have a troublshooting section that covers somethings that would typically cause the crawler to experience a 429 error.

And here’s some documentation for creating the headers in Screaming Frog as well:

• SEO Spider Configuration - Screaming Frog