Disaster recovery for expiring offline tokens: any path back after a database restore?

Rotating refresh tokens can’t be restored from a backup, and I can’t find this scenario covered in the existing threads.

Scenario

An app suffers a storage-layer failure and restores its database from a snapshot taken N days ago. Every access token in it expired long ago, and every refresh token has rotated many generations forward. The app now holds valid-looking credentials for its entire install base, and not one of them works.

This differs from the transient rotation failures already discussed: those hit individual shops during a blip and are largely covered by the 1-hour replay window. This hits every installed shop at once, and the replay window doesn’t apply because the gap is days, not seconds.

Question

Is there any supported way to recover a shop whose stored refresh token is stale, rolled back to an earlier generation?

If not, would Shopify consider a mechanism, such as the time-limited recovery key proposed in this thread?

To be clear, this hasn’t happened to us. I’m asking ahead of migrating rather than after an incident, but the answer changes what our migration and recovery plan has to look like, so I’d rather know now than find out the hard way.

This also applies for situations where Shopify’s oAuth goes down for more than an hour. (which has happened before)

I’ve been seeking answers to this for well over a month already but have been met with silence.

The lack of a critical recovery pathway is preventing many developers from migrating as when it were to occur it will bring thousands if merchants offline and cause significant revenue loss.

This is definitely very concerning, and not something I had anticipated. I really hope the Shopify team will address this.

I was planning to migrate my two biggest apps by the end of September, as anything later would be getting too close to BFCM. But at this stage, I also hope the migration deadline will be postponed until these questions have been clearly addressed and we have more certainty around the implications of the migration.

Also extremely concerning here.

These days it’s very hard to get a meaningful response from Shopify and this one is critical.

If we’re not getting a meaningful answer by early September I think we should try to find another medium to reach out to Shopify.

Thanks

My suggestion: Outside your main db, keep an encrypted append-only log of every token generation in a something like a versioned AWS S3 bucket, written before use. After DB rollback recover via log.

Also, internally we have a canary service now running performing a token refresh every minute on a development store for monitoring, before we refresh a production token we are checking the canary to ensure that shopify is responsive in past 30 minutes beforehand, to minimize risk of a 502 response and greater than 1hr timeout.

We’ve now hit a related failure in production. Not a full restore of rotated refresh tokens but close enough that the outcome is basically identical.

After we started cycling existing non-expiring offline tokens in the background (Cycle existing tokens without waiting for a merchant), some shops still looked installed but had an empty access token and no refresh token. A dump from a few days earlier still had the legacy non-expiring offline token. Restoring it and calling Admin GraphQL returned 401.

So either the cycle succeeded at Shopify and we lost the new access/refresh pair, or the old token was revoked another way. Either way the dump cannot replay it. There is no refresh grant without a refresh token. The only path left is the merchant opening the app so session-token / OAuth can mint a new pair.

That is exactly the gap this thread describes, and Shopify’s own docs admit it: cycling is irreversible, unsafe to replay, and if you lose the response the merchant has to reauthorise. There is still no supported recovery when the credential Shopify currently considers valid is not in our database.

This is a bad design for any app that runs jobs, webhooks, or App Proxy without the merchant sitting in Admin. “Reopen the app” is not disaster recovery. A Partner/App Management reissue for an existing install, or the time-limited recovery key suggested above, should have existed before you made cycling one-way and put a January 2027 deadline on public apps.

@Paige-Shopify @Billy-Shopify @Steve-Shopify @Ken-Shopify @Wes-Shopify @Liam-Shopify @Harshdeep-Shopify

Hey guys, apologies for the blanket tag, but I do want to make sure Shopify is aware of these concerns and we can hopefully get the ball rolling on making migrations a bit safer.

Ultimately, the process seems to rely on too much “working” as expected. Which, as we all know, is often not the reality. Migrating from non expiring to expiring access tokens - #29 by Levente_Kovacs puts forward a good proposal.

Whatever the right solution, we do need someway of an app developer being able to reissue a token for an install, without the merchant having to do anything.

Personally, one of my apps now has 100+ merchants that are stuck in this state and I can’t do anything about it.

Thanks

It’s absolutely insane to me that nobody from Shopify has jumped in here. I can only hope they are coming up with a plan to address this, frankly, poor migration experience.

2 more days have gone by and still nothing from Shopify :joy:

Not even an acknowledgement that it’s been looked into / discussed internally.

This just dropped on the changelog..

That’s great news! Finally a proper solution!

Happy days! What a great way to end the week!

Grateful for Shopify for this, but there does need to be more communication for things like this.

Hi all - we think the change we rolled out today will significantly increase the reliability of the token refresh system and address the concerns that have been raised on this thread and elsewhere as well. Please continue to provide your feedback as it’s been instrumental in helping us understand the concerns the developer community has about this change from non-expiring to expiring tokens. I’ve also posted about this change here with some more details.

Hi @TerenceShopify,

Thanks for shipping this. It’s a real improvement, and it closes the lost-response case properly.

But I don’t think it closes the question this thread was opened for, and I’d like to check my reading.

As I understand the changelog, retention is one generation deep, not time-indexed: the previous refresh token is kept only until the replacement is used, after which it’s retired. The 30 days is a ceiling on that, not a window you can restore into. So the effective recovery window per shop is the gap between its last two refreshes: minutes or hours for an app doing background work.

In the scenario I described, the app ran normally for days and rotated many generations. A snapshot from a few days ago holds a token that was retired the moment the next one was used. Concretely: an app restores a database backup from three days ago. How does this change help it recover API access?

If the answer is that it doesn’t, what do you recommend instead? The workaround being discussed is an append-only log of every token generation in separate storage, ideally at a different provider so it survives the same failure.

Two concerns with that as the official answer:

  • It means replicating every merchant credential into a second system, which widens the attack surface and adds a second thing to secure, audit, and get right.
  • It’s meaningful complexity that every partner running background work without merchant UI interaction now has to build and maintain.

Is that the intended expectation, or is a platform-side recovery path still on the table?

In your specific scenario (DB loss with the restored backup being several days old) there are two recovery paths:

  1. The merchant opens the app and the app acquires a new token. If the app is usually opened by the merchant on a regular basis this would be a simple recovery. If the app is not used interactively the app developer could directly reach out to the merchants with the app installed to recommend they open the app.
  2. Contact Shopify Partner Support to get help with recovery.

Obviously having to contact merchants to ask them to open an app isn’t the most desirable process, but it would be effective. Shopify support would be able to explore other recovery options with the app developer that are not self-service but could be made available in this type of disaster recovery scenario.

We have apps where merchants set it up once and then rarely open the app again. It’s not only “not the most desirable solution”, this is simply not a solution, especially given that most merchant dont even read emails sent by apps. They would contact us two weeks later, upset, after they randomly noticed that the app is not functioning and they’re losing money.

For contacting the partner support I’m doubtful they could help if you dont setup upfront a streamlined process for this.

Looking forward to a stronger resolution process

I’ve created another thread to ask Shopify to fix a gap in the offline-token migration:

When cycling legacy → expiring tokens, Shopify destroys the old token in the same request that issues the new pair. If an app never persisted the new tokens (lost response, worker crash, DB write, same class of failure they already fixed for refresh), that merchant is permanently orphaned until they reopen the app. No server-side recovery.