Please send subscription ID or Client ID with webhook payloads?

Webhooks and Events
1

Currently Shopify sends identifying information about the shopify store that a webhook payload is for, but it doesn’t send which Shopify Client ID or which webhook subscription caused the webhook to be delivered. This information is captured and displayed in the Delivery Logs in the partners dashboard, but it’s not sent in any headers.

If you have more than one Shopify application running on the same set of servers, this makes identifying which application an incoming payload is for harder! If you have an exact 1-1 mapping between infrastructure and client IDs, then you implicitly know which client ID you are, but if you have multiple, there’s no clear indicator in the payload. The reasons for having multiple client IDs around might be:

  • cost optimization, where you run multiple apps in the same process or against the same database just to save on costs for deploying more than one app
  • temporary migrations from one client ID to another
  • multitenant platforms like integrations providers or Gadget that host many Shopify apps

If you do have multiple client IDs, one way to identify which client ID caused a webhook to be sent is to check the HMAC of the payload against all the client secrets that could have possibly caused the webhook to be sent. Whichever client secret validates the HMAC is the one that sent it. This strategy is prohibitively expensive if you have many client IDs thought.

Another strategy for identifying which client sent a payload is to put the client ID in an HTTPS webhook subscription URL. This allows you to know which client sent a payload because you can create dynamic URL patterns in your application, and pass information back to yourself within that URL. I currently do this to work around this problem, with URLS like:

// for client id 1
"https://myapp.gadget.com/api/shopify/webhooks/1"

// for client id 2
"https://myapp.gadget.com/api/shopify/webhooks/2"

But, this strategy doesn’t work for the EventBridge / GCP PubSub subscriptions that Shopify recommends, because you can’t do dynamic topic patterns in these systems, just static topic names. GCP also has hard limits on the number of topics you can create total, and its expensive infrastructure-wise to subscribe to a lot of topics at the same time, so ideally we could put all our webhooks on the same topic, like:

all-webhook-payloads

and then inspect the individual payloads for which client id they are for.

Any chance we could get the client id added as a webhook header and pubsub metadata item?

2

I assume you’re asking for this functionality on behalf of Gadget’s infra for supporting partners using your platform.

For the normal use case, the app ID is an extra detail that’s implicit.

Just to play devil’s advocate, this scenario by a single Partner actually dances very close to Shopify App Store Guideline #6 - Overlapping Functionality.

I’m not saying this definitely is, but it could be interpreted that way.

If I were developing a platform for building Shopify apps on behalf Partners, I would personally just invest the time to build an ingress system based on Lambdas/Serverless Functions with a predictable URL structure that includes their app ID:

https://shopify.wh.${appId}.gadget.dev

Then you can map the incoming messages to the appropriate tenant in your platform.

Just a thought, that way you have more control rather than lobbying for this feature.

3

Indeed I am asking for Gadget!

Just to play devil’s advocate, this scenario by a single Partner actually dances very close to Shopify App Store Guideline #6 - Overlapping Functionality.

We aren’t duplicating Shopify’s functionality – we’re using exactly the primitives they offer to help app developers build apps for their ecosystem. We’re not replacing other apps, or replacing Shopify, we’re a hosting provider that knows about Shopify webhooks and makes it easy to receive them.

Lambdas/Serverless Functions with a predictable URL structure that includes their app ID

This is exactly what we’ve done, as I said in the original post, but it is not the best practice. Shopify explicitly recommends using GCP PubSub, and I agree with their recommendation. Decoupling consumer and producer throughput is a big stability win, especially for BFCM, and PubSub is more efficient for both them and for us as you can use batch consumers and you don’t need to verify HMACs. We’d like to adopt the best practice for these reasons, but can’t because of the aforementioned limitations around topic cardinality in GCP or EventBridge.

So, if anyone from Shopify is listening, would you be open to adding that header?