Webhook Recovery After Outage

Webhooks and Events
1

I have a question about webhook recovery after an outage.

If an outage occurs and events are successfully processed during that time, will the webhooks for those events be sent after the outage is resolved? Or will those events be lost?

Thank you for your assistance.

2

Hi @Shota_Yamamoto,

I think it can vary depending on the type of outage but here’s what we’ve seen on our app that handles millions of webhooks/day.

Generally if there’s an outage on Shopify’s side we’ll see the webhooks be sent after the outage. The few times this happened (last night being on of them), the amount of webhooks we receive drops significantly during the outage, but then there’s a surge of messages afterwards as all the delayed messages get sent. Depending on the volume of webhooks you’re receiving, this surge could end up causing an outage on your end.

For outages on your end, Shopify will retry sending the webhook message 8 times over a 24 hour period. The important thing here is to keep track of any updated value in the payload, or the X-Shopify-Triggered-At HTTP header because you don’t want to process the retried webhook out of order.

Lastly, while the webhooks are fairly reliable, it’s good to have a background job that runs which does a full re-download of the data from the Shopify API. You can run this on a schedule, or after an outage, to make sure that your local data is in sync with Shopify’s

Best,
Daniel

3

@Daniel_Ablestar Thank you for your reply. I’m relieved to hear that it will be resent. I will prepare a system to check whether the data is consistent, but I will wait for the resend.

4

@Daniel_Ablestar If there is any documentation that clearly states that this will be resent, please let me know.

5

I don’t see any documentation that states they’ll be resent, I just have the anecdotal evidence that we see a big surge in webhook deliveries after an outage like yesterday’s. If you’re looking at yesterday’s outage and still haven’t received a webhook message I would check the Partner admin webhook stats to make sure that it doesn’t show up as a failed delivery.

If it doesn’t, the message is probably not going to arrive so I would look to building/running a job to reconcile your data with Shopify’s as Shopify recommends.