Webhook delivery delays?

Partner Discussions
1

I am still learning the ropes with writing a custom Shopify app. For the most part I have it working. My app receives webhooks and processes them accordingly. However, I have another app I use for testing where it creates about 125 products via GraphQL api calls. I have built-in delays, so those 125 take about 10 minutes to complete. During that time, my main app is receiving the webhooks and processing correctly. However, it seems after a while the webhook delivery is “paused”. In my case I stopped receiving webhooks around my 95 item. This pause lasted about 5 minutes, and then another batch of webhooks came over giving me maybe another 10 items. Here it “paused” for 20 minutes before receiving another small set. Am I doing something wrong or not doing something that is causing these delays?

2

Hi @John_Miller

It doesn’t sound like you’re doing anything wrong - what you are experiencing is a common scenario when dealing with Shopify webhooks and high-volume API activity. Here’s what’s likely happening and how you can address it:

  1. Webhook Delivery Throttling and Retries
  • Shopify expects your app to respond to webhook deliveries within 5 seconds. If your app takes longer, or if your server is slow to respond, Shopify will consider the delivery failed and will retry up to 8 times over 4 hours. Each retry increases the delay between attempts, which can cause the “pauses” you observe.
  • If your app is busy processing a large number of webhooks (such as when 125 products are created in quick succession), it may not be able to keep up, leading to delayed responses and triggering Shopify’s retry/backoff mechanism.
  1. API Rate Limits
  • Shopify’s Admin GraphQL API uses a calculated query cost system with a leaky bucket algorithm. If your app or another app on the same store is making a lot of API requests, you may hit rate limits, which can slow down both API calls and webhook processing if your app is making synchronous API calls in response to webhooks.
  1. Webhook Debouncing
  • Shopify may debounce (drop) webhook deliveries if the payloads are identical and occur within a short time window, especially if you have customized your webhook payloads to only include non-unique fields. Always include a unique field like updated_at to avoid this.

Best Practices to Avoid Webhook Delays

  • Respond Quickly: Always respond to Shopify’s webhook POST with a 200 OK as fast as possible (ideally within 1 second, but no more than 5 seconds). Do not do heavy processing synchronously in your webhook handler. Instead, enqueue the work for later processing and immediately return a 200 OK.
  • Queue and Process Asynchronously: Use a message queue or background job system to process webhook payloads after acknowledging receipt.
  • Monitor Webhook Metrics: Use the Partner Dashboard’s webhook metrics to monitor delivery failures, response times, and retry rates.
  • Implement Reconciliation Jobs: Periodically fetch data from Shopify using the API to ensure you haven’t missed any events due to webhook delivery issues.
  • Handle Rate Limits: Implement retry and backoff logic in your API calls, and avoid making excessive API requests in response to webhooks.

Troubleshooting Steps

  1. Check Your App’s Webhook Response Time: If your app is taking longer than 5 seconds to respond, Shopify will delay or retry deliveries.
  2. Review Webhook Delivery Logs: In the Partner Dashboard, check the webhook delivery logs for failed deliveries, high response times, or excessive retries.
  3. Optimize Processing: Move any heavy processing out of the webhook handler and into a background job.
  4. Check for Rate Limiting: If you’re making API calls in response to webhooks, ensure you’re not hitting Shopify’s API rate limits.

Hope this helps and best of luck with the rest of your custom app build :slight_smile:

3

Thanks for the information. This is very helpful. In my production region, I do use AWS EventBridge to capture webhooks and it may be working fine there - at least I haven’t seen any issues. This was on a local dev environment where we implement things a little different as some of these machines are not connected to AWS. We use ngrok to direct the webhook traffic to a locally running API endpoint (ASP.NET Core). I haven’t seen any errors in the ngrok log, but maybe ngrok isn’t receiving fast enough and Shopify thinks its busy? I compared logs between ngrok and my API. The API is processing everything ngrok sends it - and all within milliseconds. So maybe it is just an ngrok issue not getting all webhooks and forwarding in a timely manner?