Cancellation Request: Concurrency Issue?

[Theodor_Freund_at_HT]

Hi @all,

I am having a weird issue.
I wrote a fulfillment service API endpoint , it runs in google cloud. I noticed a strange behaviour when using it with the shop that I cannot quite explain:

I noticed that the fulfillment_request goes through without any issues, but the cancellation_request does not process properly, even though they basically do the same thing.
When I start the request via Postman, it always works.

After adding a ton of debug messages I think I finally found the problem:
When I receive the cancellation request, I do a query towards shopify-api. It uses the “assignedFulfillmentOrders” graphql query. This query works in local debug as well as in the cloud when triggered from postman.
When shopify sends the cancellation request however, the return of the assignedFulfillmentOrders query is an empty array and no processing happens.
When I then cancel a second fulfillment order that is accepted by this fulfillment service location, I suddenly find the first pharmacy order in the logs and processing happens for that one, but not the second.

What am I missing? Could it be that the cancellation request is posted towards the callback of the fulfillment service location before the actual cancellation is done in shopify?

tl;dr Is there a concurrency issue between the callback invocation of a fulfillment service and the status of the fulfillment order when requesting cancellation?

[ipvalverde]

Hi @Theodor_Freund_at_HT , thanks for reaching out!

Just want to confirm I get it right:
You’re firing a GraphQL request to load the fulfillment orders assigned to your Fulfillment Service App. So you’re using your app’s access token and sending a request with a payload similar to the following:

query getAssignedFulfillmentOrders {
  assignedFulfillmentOrders(first: 10, assignmentStatus: CANCELLATION_REQUESTED) { ...  }
}

But if you call it right after a “cancellation request” callback request, it might not contain the fulfillment order you’re looking for. Is that correct?

After investigating the code I can confirm that this behaviour can occur when the callback request is sent very quickly. Given the way we manage the state, aiming for upmost consistency in our data, we first send the webhook events and, only if they succeed, we start the process of updating the fulfillment order status in the database. This means that we can have a small delay between the time you receive the callback event and when the status is updated.

Consider one of the following approaches to address it:

• If possible, add a small delay in your app from the time you receive the webhook to when you call the assignedFulfillmentOrders GraphQL query. This is likely a background job in your system that shouldn’t affect the user experience;
• Use webhooks, which are a more robust alternative to fulfillment service callbacks. The webhooks are fired after all the data has been updated and you can also get extra data from it, possibly removing the need for an extra request to assignedFulfillmentOrders in your flow. You could create a webhook subscription for the cancellation request event with the following payload:

mutation subscribeToWebhook {
  webhookSubscriptionCreate(
    topic: FULFILLMENT_ORDERS_CANCELLATION_REQUEST_SUBMITTED,
    webhookSubscription: {
      callbackUrl: "<WEBHOOK_CALLBACK_URL_GOES_HERE>",
      format: JSON,
    }
  ) {
    userErrors {
      field
      message
    }
    
    webhookSubscription {
      id
    }
  }
}

[Theodor_Freund_at_HT]

Hi @ipvalverde ,
thank you for the swift reply.
This almost answers all my questions, but I still have a couple left:

@ipvalverde

But if you call it right after a “cancellation request” callback request, it might not contain the fulfillment order you’re looking for. Is that correct?

Correct.

@ipvalverde

If possible, add a small delay in your app from the time you receive the webhook to when you call the assignedFulfillmentOrders GraphQL query. This is likely a background job in your system that shouldn’t affect the user experience;

I think I can do that. What sort of timedelta would you suggest? As this is tied into the merchant feedback, I would like to keep it to a minimum.

@ipvalverde

Use webhooks, which are a more robust alternative to fulfillment service callbacks. The webhooks are fired after all the data has been updated and you can also get extra data from it, possibly removing the need for an extra request to assignedFulfillmentOrders in your flow. You could create a webhook subscription for the cancellation request event with the following payload:

I mean, yes I could, but I was under the impression, that I followed the documentation to the letter to explicitly avoid such issues: Fulfillment Service Docs

I don’t really mind to use webhooks, but now I am asking myself if the same issue happens with the “FULFILLMENT_REQUEST” api call as well, maybe?

In any case, I am unsure as to how the webhooks differ from the api call,.

@ipvalverde

Given the way we manage the state, aiming for upmost consistency in our data, we first send the webhook events and, only if they succeed, we start the process of updating the fulfillment order status in the database.

If that holds true, wouldnt it be the same for the other webhooks as well?

We just built all this for our new page and we soft-launched yesterday, so I would love to get these things implemented in a future-proof way before traffic pics up

[ipvalverde]

@Theodor_Freund_at_HT

I think I can do that. What sort of timedelta would you suggest? As this is tied into the merchant feedback, I would like to keep it to a minimum.

The recommended approach is to keep pooling the data until it become available. You could add a wait of a few seconds and then try to consume it, repeating the process until the data shows up.
This would add more resiliency to your system, since we could have a scenario with higher delay time due to increased traffic for the shop.

@Theodor_Freund_at_HT

I mean, yes I could, but I was under the impression, that I followed the documentation to the letter to explicitly avoid such issues: Fulfillment Service Docs

I don’t really mind to use webhooks, but now I am asking myself if the same issue happens with the “FULFILLMENT_REQUEST” api call as well, maybe?

In any case, I am unsure as to how the webhooks differ from the api call,.

You’re right, the recommendation is still to use the callback since this was implemented even before the webhooks. We do not have plans in our roadmap to remove the current fulfillment service callback implementation, and if we were do it, it would go through a deprecation cycle of at least one year.

The webhooks are more versatile since you can use them for different topics, it also has a delivery metrics report, you can also receive a payload with data as part of the request which might remove the need of making additional request to retrieve data. But it’s up to your strategy if it makes sense or not to use it.

@Theodor_Freund_at_HT

If that holds true, wouldnt it be the same for the other webhooks as well?

We just built all this for our new page and we soft-launched yesterday, so I would love to get these things implemented in a future-proof way before traffic pics up

The implementation of the “fulfillment request” is different from the “cancellation request”. The “fulfillment request” will not present the same error - instead, it will first update the fulfillment order state and used a background job to send the HTTP callback request to the fulfillment service. That’s why in the documentation, we only suggest to add a pooling logic for the “cancellation request”.

After further investigation, I would go back on what I said earlier . The “fulfillment request” logic sends the callback request as a background job to deal with the high level of errors from fulfillment services apps when handling our requests.

[Theodor_Freund_at_HT]

Allright. Thank you so much for all the details and research.

I have one last question then:

If I am going to switch to webhooks completely:
Do I still have to have the API for the callback running to give back a 200 so that the merchant view is working correctly?
(so is there a status for the fulfillment order in between the click on “request cancellation” and the 200 return from the API or can I just skip it completely)

[ipvalverde]

@Theodor_Freund_at_HT

Do I still have to have the API for the callback running to give back a 200 so that the merchant view is working correctly?
(so is there a status for the fulfillment order in between the click on “request cancellation” and the 200 return from the API or can I just skip it completely)

You would still need to set a callback URL when registering your fulfillment service returning 200. Also, that callback is still useful if your fulfillment service provides tracking support or inventory management.

[Theodor_Freund_at_HT]

@ipvalverde Sorry, let me rephrase that.
Do I still need the /fulfillment_order_notification endpoint to return 200? I do understand that the others are still useful.

And another Thank You for the effort!

[ipvalverde]

@Theodor_Freund_at_HT

Do I still need the /fulfillment_order_notification endpoint to return 200? I do understand that the others are still useful.

Sorry I was not very clear on that. Yes, you still need the endpoint returning 200 for the fulfillment_order_notification requests. Maybe this will be the biggest consideration on whether or not using webhooks instead of callbacks since you still need to keep an endpoint for callbacks (even though it can be a dummy one).

And thanks for using the Shopify platform @Theodor_Freund_at_HT !