Hi Shopify Dev team 
We’re building a public app with Checkout UI Extensions (rewards blocks shown to logged-in buyers) and Customer Account UI Extensions (profile, order status and dashboard blocks). We run a Playwright end-to-end suite against a development store on every pull request and before every deployment.
The flows that need a logged-in customer keep getting stuck on Shopify’s verification screen during sign-in, so we can’t test them reliably in CI. We’d like to know the recommended, supported way to automate these flows.
1. What is the cause?
As far as we can tell, the new customer accounts sign-in (shopify.com/authentication/…, email plus one-time code) is protected by bot detection: Cloudflare with hCaptcha. A browser driven by an automation tool is flagged as non-human. It gets worse when sign-in is repeated, for example each test run, or a test retry that sends the same email again within a few seconds. We see two responses:
- “Your connection needs to be verified before you can proceed”, shown either after clicking “Sign in” or after submitting the one-time code (sometimes as a redirect loop on /oauth/authorize).
- “Couldn’t sign you in. Retry or check your connection.”, shown right at the email step once the account has been challenged a few times.
Once triggered, it can keep happening for hours for that test customer.
2. What is blocking the automation flow?
To test our extensions as a logged-in buyer, the test has to: add a product to the cart, go to checkout, sign in, and then check our extension blocks. Sign-in is the step that fails:
- The first attempt sometimes works. Later attempts on the same day are challenged and the test hangs or times out.
- We can’t solve or bypass a CAPTCHA, and we don’t want to.
- Saving a signed-in browser session between runs hasn’t been dependable, because the challenge can still appear when the session is reused.
- As a result, the checkout tests for logged-in buyers and the Customer Account extension tests are either flaky or have to be skipped, which leaves a real coverage gap for features merchants rely on.
3. Our automation tech stack
- Playwright (TypeScript) running headless Chromium, with the Page Object Model and custom fixtures
- GitHub Actions on a self-hosted Linux runner (one shared test identity, tests run one at a time)
- App: a React Router app using @shopify/shopify-app-react-router, with Prisma and PostgreSQL
- New customer accounts sign-in (email + one-time code). We read the code from a dedicated test inbox through the Mailtrap Inbound API.
- Allure and Playwright HTML reports
4. What we’ve tried so far, and what each is for
| Approach | Use case | Result | |
| Admin side: session token signed with @shopify/shopify-api/test-helpers getJwt() plus a real stored offline session, sent straight to our app URL | Testing our embedded admin pages without the admin.shopify.com login, which is also behind Cloudflare |  Works well |
|
| Storefront password unlock with the store password form | Getting to the storefront and cart for guest checkout tests | Works |
|
| Guest checkout tests (no sign-in) | Checking our checkout extension’s guest/opt-in messaging | Works |
|
| Real email + one-time-code sign-in in Playwright, with the code fetched from the Mailtrap API | Testing checkout as a logged-in buyer |  Works now and then; challenged often |
|
| Turning off test retries and adding long random back-off between attempts | Avoid triggering the challenge with rapid repeat sign-ins | Fewer challenges, still not dependable |
|
| Saved browser session (Playwright storageState) shared between runs | Sign in once and reuse it |  Dropped. The challenge can still appear, and session lifetime isn’t predictable |
|
| Pre-check through our backend (skip sign-in when the customer isn’t eligible to see the feature) | Avoid unnecessary sign-ins | Fewer sign-ins, but doesn’t help the flows that need a signed-in buyer |
|
| Currently trying: a Customer Account API access token (confidential client on a Headless storefront, one manual human sign-in to get a refresh token), set as buyerIdentity.customerAccessToken on a Storefront API cart, then opening cart.checkoutUrl | Open a logged-in checkout without the hosted sign-in page |  In progress, not yet verified |
Our questions
- What is Shopify’s recommended / supported way to run automated E2E tests of logged-in checkout and Customer Account UI Extensions on a development store?
- Is the Customer Account API token + buyerIdentity.customerAccessToken + cart.checkoutUrl approach the intended path for checkout? Does it also work for Customer Account extension pages (Profile / Order status), or do those always need the hosted sign-in?
- Is there a test mode, allowlist or dev-store setting that relaxes bot verification for development stores or known CI environments?
- Do the shopify app dev preview tools or any other official option let us render Customer Account / checkout extensions as a test customer without a real sign-in?
Thanks a lot for your time. Any pointers to docs or examples would be really appreciated!
AnnexCloud Team