Has anyone been able to whitelist IP’s successfully? We are using a technical crawler that is getting intermittently blocked by Shopify, and to rectify this on any other platform we’d whitelist. However, Shopify support has said they currently don’t support whitelisting. Any ideas?
Hi
We have the same problem.
Is it possible to whitelist an IP ?
Regards
Just curious, why scrape your own Shopify site? Since you’re the owner of this site I presume, can you just use the Admin API to retrieve the data about products you need?
Hi Dylan, thank for your comment
Crawl your website is very useful.
For exemple we do non-regression tests each hour.
I want to recieve an alert if one of my TITLE change, an URL etc… very fast to fix before for example Google notices the change
Also we do each month full crawl and comparison to find broken links, links to 301/302 etc.. and to analyse the evolution avec the website
That’s why I’d like to be able to whitelist the IP of our crawler
Got it - that makes sense.
I’m not sure what you’re using for your E2E suite, but there may be an option to throttle your requests to help prevent bot mitigation.
Additionally, there are tools out there that help mock a browser as closely as possible to prevent detection. One such tool is browserless.io, which you can connect to, and they take care of running the browser instances and captcha/cloudflare prevention.
thank you for your reply and idea
We use https://tool.oseox.com which is great
A lot of users crawl their own site and unfortunately the IP rules seem to be very strict since 2-3 weeks (excluding Google IP of course…)
Without whitelist it’s impossible
Services like Cloudflare and other solutions (cache, anti-bot, etc.) enable whitelisting. I hope a shopify developer will read this topic and add this feature on the roadmap ^^