Any guidance how to handle login to Shopify for automated UI testing of Shopify app using Playwright, Cypress etc?
We wish to run this in our CI servers, but the frequent login confirmation pages or cloudflare captcha (even after first successful authentication) is causing flaky behavior).
How are you handling automated browser testing of your apps?
Unfortunately I don’t think there is a good way to do automated UI testing for embedded Shopify apps. We looked into this a while ago but ultimately decided to pass on trying to do it.
Like you mentioned, if you try to access the Shopify admin with an automated tool you’ll often get a Cloudflare captcha, especially if you’re trying to access the admin from a datacenter IP. You might be able to temporarily get around this by persisting the browser session, and manually logging in with that session to complete a captcha when it appears, but it’s not a stable solution.
Additionally I’m not sure if automatically accessing the Shopify admin, even to test an app, is permitted. Section 1.9 of the Shopify terms of service says that:
You agree not to access the Services […] using any robot, spider, scraper, or other automated means.
The only possible solution I could think of to get it to run reliably would be to make your own HTML page that loads up your app in an iframe. I don’t know if this is even possible though, you’d have to replicate a bunch of the app bridge and session authentication features.
I came across this problem before Unable to run automated test in Shopify app in embedded mode reliably · Issue #222 · Shopify/shopify-app-bridge · GitHub. Most devs just seem to login to playwright and run the test manually unfortunately there is no way to run this in the CI
We do need a way to mock app bridge reliably to run the E2E in CI
Looks like there is a PR to run Playwright test now Add End-to-End Testing Infrastructure with Playwright by IliasHad · Pull Request #986 · Shopify/shopify-app-template-remix · GitHub 
Don’t forget you’d also need to disable 2FA auth on your Shopify account, which is a security issue.
It might be possible to create a staff account with limited permissions to run these tests on behalf of. But using your main partner account for E2E tests would be an issue because of 2FA.