This is an ongoing issue that started about 1 year ago, as soon as we migrated from AppBridge 4.x.x to the CDN one. It never happened before that, as far as I can remember.
Just about every single day, we see a few requests that come with no JWT. i.e. the token is undefined. We spent a few weeks collecting request IDs, timestamps and screenshots to build a “case” and sent to Shopify, but the issue persists to this day.
I’m not sure if the issue is from AppBridge’s fetch implementation, or something with the redirects in the Admin. For example, a merchant clicks an admin action, it directs to the app, with the token in the param, but the value is undefined. The same happens with fetch requests.
Hey @flavio-b - happy to take a look into this for you. Just out of curiosity, was this the same issue we were looking at related to the expired JWT tokens, or a separate issue where they’re just being sent as “undefined” completely?
If you have a Support Ticket ID, I can take a look at this specifically and see if we can get this looked into further. I haven’t seen any other reports of this behaviour on my end, but more than happy to investigate for sure.
Before, I was noticing more expired JWTs. Nowadays, I’m seeing more undefined tokens.
Sometimes once a day, sometimes half a dozen.
It’s not much, but I think there may be a specific bug with token logic. For example, we never see other params/headers missing. It’s always the token.
At this point I was just hoping to see if someone else would confirm this, so we can rule out a bug in our implementation. Although I can’t see what we’re doing wrong, because we don’t have any code that deals with tokens directly, right? They’re appended to requests by Shopify, via param or fetch, and handled by the Shopify App middleware.
Hey @flavio-b, thanks for clarifying! Would you be able to capture and share a complete request payload when the token comes through as undefined if possible?
If we can see the full request headers (especially any Authorization, Referer, Origin, and X-Shopify-* headers), the full request URL with parameters where the token shows as undefined, and any browser console errors that appear at the same time we can narrow down what’s causing it.
If you could also share how you’re initializing AppBridge with the CDN version and grab some network timing info about when these requests happen, that would give us a complete picture. The new version of App Bridge does handle tokens differently than the older npm package versions, so I am wondering it it’s unique to the new version.
The fact that it’s always the token that’s missing (and not other params/headers) is definitely interesting - hope to hear from you soon, happy to investigate as always.
We have similar issue because JWT is invalid.
Exact error is: NotBeforeError: jwt not active.
So why is appBridge sending an expired token, and doesn’t generate a fresh one while the user is still logged in shopify admin, and making calls to the app backend?
In the next request it uses a working token, then after few minutes it will fail again on the next call. A bit frustrating.
I won’t have all the details, because the issue happens in the merchant’s browser. All we see is the params that hit the server, where the token is undefined.
It’s not something I was ever able to reproduce in my browser. It affects different merchants at different times throughout the week, randomly.
We initialize AppBridge from the CDN, with <script src="https://cdn.shopify.com/shopifycloud/app-bridge.js"></script>
We just got the first undefined token of the week. This is all the information I have on this end. The Request ID will not be useful to you, so I’m not sure if this helps at all.
Those logs that you see from the missing token come from ShopifyAPI::Auth::JwtPayload#decode_token, which is a method from the Shopify API gem that we override just to log the token that is passed in to the Shopify middleware.
Thanks @flavio-b - I’ll raise this with the team and we’ll look into things for you. I can’t guarantee a turnaround time as always, but I will keep you up to date in the thread here as soon as I have more info.
I appreciate that. I hope they can find something.
We’re approaching the one year mark since switching to CDN AppBridge and seeing these errors. It sets off daily alarms in our monitoring, so it’s getting really tiring and distracting.
I did another search in the logs today and found that there are indeed many instances of expired tokens. So that issue continues, not just undefined tokens.
There’s something deeply weird in the token logic, as far as I can see. Is there any chance of having an open source version of the CDN AppBridge, so I can do some diffs vs AppBridge 4, and see if we can find the issue?
If not, can you share the part that performs token fetching, expiration check and sets the header?
No worries @flavio-b. When it comes to open-sourcing App Bridge (or just an open source version of it), that would be up to our product team themselves. My understanding is that we don’t have plans to release AB to open source at the moment, but I can definitely pass along the request.
I was able to dig into this with the team, but we’d require a bit more info to investigate this. Are you able to replicate the missing token issue when you try to interact with your app in App Bridge? If you’re able to replicate the same issue, could try recording a HAR file when you do? (More info here).
If it’s helpful as well, I can try replicating things on my end and share my test shop info with you (if you’re able to share your app ID with me - I can set up a DM with you and we can go from there to troubleshoot further) - let me know if I can clarify anything as always
Unfortunately, I’ve never need able to replicate it, even after a year. However, as @Aisbjo might be seeing something similar. Can you provide some details to Alan?
I’ll use some AI to deobfuscate the token logic a bit and see if anything comes up.
Hi Alan, I’m looking at a deobfuscated AppBridge token logic, and found a few potential issues (of course, variable names and formatting will differ from your source code).
The part that sets the header does not check whether the token is undefined or expired:
if (shouldAddAuthToken) {
request.headers.set("Authorization", "Bearer " + await api.idToken());
}
The function that provides the token has no error handling or retry mechanism in case the Promise is rejected. It can return undefined.
A server can append a special header and trigger a re-fetch, but this requires a roundtrip to the server only for it to determine whether the session is “invalid”. Still, it ultimately calls the function above again, so there’s no guarantee the re-fetch will work. The promise could reject again and this would not fix that.
I think that, given the token “factory” is running in the context of the client, the client-side code should be doing more to ensure a valid token is sent, avoiding the roundtrip. Of course, the server has to validate sessions eventually, but this might be hiding the fact that the token source is failing more often than you might be aware of.
It makes the server responsible for telling AppBridge that the token is undefined or expired, which are two things it could have already checked for.
Another problem is the retry mechanism based on X-Shopify-Retry-Invalid-Session-Request is not super documented, at least on the shopify_app/shopify_api gem side. There’s a presence of this header deep in one of the gem files, but I tried passing an expired token to my app and it didn’t trigger that code path at all. It’s not clear how to implement that mechanism, and perhaps one should not have to play with that. It belongs to some authentication middleware anyways.
I hope this helps shed some light on the potential issues!
Thanks for the details on the deobfuscated App Bridge code there, @flavio-b, this is super helpful. I appreciate you taking the time to dig into the token handling logic.
Without being able to reproduce the issue or capture a HAR file when it occurs on our end here, it’s a bit challenging to pinpoint the exact cause, but your findings about the missing error handling and retry mechanisms might help with digging into this further.
The fact that both you and Aisbjo are experiencing similar issues with undefined and expired tokens suggests there may be an underlying issue, but I’m still unable to say for certain.
I’ll bring this information along with your code analysis to our product team for further investigation though, since it could help explain why these intermittent failures are occurring.
I still can’t guarantee a specific timeline for resolution, but I’ll make sure the team is aware of the impact this has been having on your monitoring and daily operations over the past year for sure and loop back with you with any info I can share.
Hey @flavio-b - I was able to work with our product team on this, and we’ve been your findings (thanks again for these!) about the missing error handling in idToken() and we do align with the assessment that undefined JWT tokens shouldn’t be happening (expired tokens are expected behavior, but undefined ones are not).
To help us investigate further with the App Bridge team, could you capture some error logs that include Shopify request IDs when these undefined tokens occur? Definitely realize it’s not ideal since this appears to happen intermittently, but our product team did let me know this would be the most efficient way to track these and would give the team concrete examples to trace through their systems.
I also have an update on the “jwt not active” errors that @Aisbjo mentioned. Our team mentioned that this seems likely due to a clock difference between your server and ours.
To resolve this, you’d just need to add a 10-second leeway when verifying JWTs - here’s an example from our JS library:
Most JWT libraries should support a clockTolerance or similar parameter.
Once I hear back from you with the examples, I’ll be escalating the undefined token issue to the App Bridge team with your code analysis and any logs you can provide there .
I still can’t guarantee a timeline, but having specific request IDs will definitely help them investigate more effectively. In the meantime, implementing that JWT leeway could potentially resolve some of the expired token errors you’re seeing, so I wanted to mention this as a possible workaround for some instances of this. Hope to hear from you soon and thanks again for your patience as always
About the expired JWTs, the Shopify API gem already includes some leeway, but we’re seeing tokens expired for longer than that. I quickly checked one random token now and it was expired by about 22 seconds.
We’ve already increased the leeway significantly a while ago, otherwise it would have been almost impossible to manage, as we’d be getting dozens of “expired JWT” alerts daily, on top of the “undefined” ones.
In terms of the server clock, I’m not sure… the apps run on Heroku/AWS, and they’re pretty up to date. I haven’t noticed any unexpected strangeness when dealing with database timestamps and log timestamps, and I do check those a lot, for other reasons.
Just to get some clarity on the exact Shopify Request ID that they need, I assume they want the Request ID that comes in response from AppBridge fetching the token from a Shopify token provider backend, correct?
Hey @flavio-b, thanks for the ping and for confirming the ~22s expiries you’re seeing there.
For your question regarding the request IDs, I believe these would be the session tokens that are delivered via postMessage from the Admin, so there’s no Shopify exact request ID tied to token issuance. I think what we’re after is a HAR that captures both the top-level Admin frame traffic (admin.shopify.com or shopname.myshopify.com/admin) and the exact iframe request that hit your backend with Authorization: Bearer undefined, including the X-Request-ID (and CF-RAY if present), full URL, headers, and timestamps.
As a mitigation, I think you could safely bump JWT clock tolerance to around 30s on your verifier, that might help reduce the “not active yet” noise you’re seeing. If you’re comfortable, I can set up a DM with you so we can work on this more one on one though, just let me know. Once we have those HARs and IDs, I’ll get back in touch with the App Bridge team and keep you updated, let me know if I can clarify anything in the meantime as always!
That’s the root of the problem: Due to the random nature of this issue, I can’t replicate and capture this data on my machine, and there’s no way to capture all that detail from merchant’s machines in production. The browser won’t allow it, plus it could violate a bunch of privacy regulations.
The only thing I could do is perhaps override one or more functions of AppBridge, to collect some headers that AppBridge already has access to in the iframe context, but as you said, this probably won’t help because there’s no Shopify request ID.
So, I don’t know what to do. My surprise and frustration is that it seems like AppBridge is not doing a verification of the token while building the header. It leaves the door open to potentially sending undefined and expired tokens.
That being said, I did not notice a single undefined token in the last ~4 days, which is very surprising and refreshing, but don’t know how long it will last. Maybe some token infra layer got updated?
Hey @flavio-b, I totally understand, capturing HAR files for intermittent issues isn’t super feasible in this circumstance for sure, especially when it involves merchant browsers and privacy considerations.
I really appreciate all the investigation work you’ve done here though, especially the code analysis identifying the missing error handling in idToken().
I’m going to share your latest feedback with the product team as well to see if there was potentially an update on our end that resolved things.
While I can’t promise specific timelines,I’ll make sure to emphasize the impact this has had on your daily operations and monitoring. If we need any additional information or has questions about your findings/just to confirm next steps, I’ll circle back to you on my end here.
Thanks again for your patience with this, and for the investigation work. It really does help make the case for improvements, so it’s super appreciated.
I spoke too soon. After 4 days with no undefined tokens, we saw one this morning. It looks like it came in the query param after the merchant clicked an Admin Link, so not an AppBridge token in this particular case.
Thanks for confirming it’s still occurring @flavio-b, I can’t guarantee we’ll be able to retrieve logs for that on our end, but since it was so recent, we may be able to look this up, would you happen to have the shop ID and the timestamp in UTC?
- happy to take a look into this for you. Just out of curiosity, was this the same issue we were looking at related to the expired JWT tokens, or a separate issue where they’re just being sent as “undefined” completely?
