How is checkout response time measured for CCS callbacks?

Built for Shopify
1

Hi everyone,

I’d like to ask how response time is actually measured for CCS callbacks in checkout.

We’ve noticed our callbacks are reported at around 500ms, but according to our own logs the full request never takes longer than 200–300ms — most requests are around 100ms server-side. So there’s a consistent gap between what we measure and what checkout reports.

Could this be caused by server location / network latency? Our servers are hosted in Europe.

I’m struggling to narrow it down further, mainly because we have no way to inspect this from the checkout side — the request doesn’t surface anywhere we can see it, and there don’t seem to be any logs for it (e.g. nothing in the Partner Dashboard / Dev portal).

A few specific questions:

  1. Does the reported response time include network round-trip (Shopify edge → our server → back), or is it purely the time our endpoint takes to respond?

  2. Is the measurement taken from a specific Shopify region, and could EU-hosted servers be adding latency depending on where the request originates?

  3. Is there any way to see per-request timing or logs for these callbacks anywhere (Partner Dashboard, API, etc.)?

Any insight into how this metric is calculated would help a lot. Thanks!

3

Hey @Jan_Ther :wave: thanks for reaching out.

These are great questions, and here’s my current understanding:

  1. It’s my understanding that the reported response time is the network round-trip and not just the time your endpoint takes to response which likely explains the gap you’re seeing
  2. You can reference our docs on Checkout performance which have some information on this. Part of your overall response time is latency between your app server and Shopify servers. The impact of latency varies between regions. Shopify servers are hosted on Google Cloud. Different levels of latency apply by region. To learn more, refer to the Google Cloud inter-region latency matrix
  3. While our carrier service metrics are currently available in the Partner Dashboard at the https://partners.shopify.com/{org id}/apps/{app id}/carrier_services URL, they are aggregates. Per-request timing or logs is a great candidate for a feature request which I can definitely submit for you. If you have any other specifics for how this would help you out, please share them here.

If you have any specific requests you are looking for more context on, I would definitely reach out to Support at help.shopify.com, create an authenticated support ticket and share the Trace ID(s) of the requests you are looking for additional information about. Our Support team should be able to help you out with this.

I’m also going to do some more digging on my end and follow up with you here when I have more detail to share. Let me know if you have any other questions in the meantime!

4

Hi, thanks a lot for the input and the advice - really appreciate it.

Per-request metrics in the dev dashboard would be fantastic, in the same way we already have them for GraphQL, REST, Webhooks, etc. Please do submit that as a feature request. The main reason it matters: the feedback loop right now is painfully slow. We made some changes that only showed up in the graph 3-4 days later, and in that window the numbers actually got worse (even though we’d added caching for deduplication). So now we have to roll the change back to confirm whether it was the cause - which means waiting another X days for data. Per-request timing would let us validate a change in minutes instead of days.

One question on the aggregation: does the reported average include requests that return an empty rates array, or otherwise very fast requests? We have hundreds of requests completing in single-digit to low tens of milliseconds, and only a few dozen in the 200-500ms range, yet the dashboard aggregate currently sits around 600ms. That doesn’t add up for me, even accounting for the network round-trip. I might be overlooking something, but I can’t figure out what.

Relocating our server would be a last resort - a latency lookup puts us at roughly 20-100ms to Shopify, so that alone shouldn’t explain the gap.

Thanks again, and looking forward to whatever you turn up on your end!
Have a great day

Jan

5

+1 on this. We’re in a very similar position.

There is consistently around a 500ms gap between our internal carrier service response times and what Shopify reports in the Partner Dashboard. Internally, our p95 is comfortably below the Built for Shopify threshold, but the Shopify graph appears to show a much higher number, with a 3 to 4 day lag before changes are reflected.

That makes incremental improvement extremely difficult. We can make a change, but then we have to wait several days to understand whether it helped, had no effect, or made things worse. Given this metric is a hard requirement for Built for Shopify, I’d expect the tooling around it to be more comprehensive and actionable.

Per-request timing would be extremely helpful here, particularly if it showed the Shopify-observed duration, our app’s response status, whether rates were returned, and ideally a trace/request ID that we could correlate with our own logs.

@Wes-Dev-Shopify Could you also reach out to the appropriate team to clarify where Shopify is measuring carrier service response time from?

Specifically:

  • Is the measurement taken from Shopify edge locations, or from a single Shopify host region?

  • If it is regional, which region or regions are used?

  • Is the reported timing based on the full Shopify-to-app-to-Shopify round trip?

  • Are empty rates array responses included in the aggregate in the same way as responses with returned rates?

Without that context, it’s very hard to tell whether we’re looking at app processing time, network latency, regional routing, aggregation behaviour, or something else entirely.

6

Hey,

We changed a few things on our side: caching everything we reasonably can, resolving the location from the address via two providers in parallel (fastest one wins), and optimizing the MongoDB query. Per the dashboard graph the result is roughly the same or even slightly worse, but going by the raw data we collect ourselves it looks good.

Here are our own measurements over the last few days (average across all requests):

Date    Requests (above 100ms)   avg (all)
19.6    216                      31ms
20.6    228                      16ms
21.6    232                      18ms
22.6    220                      20ms
23.6    197                      22ms

Even adding the round-trip to our server (roughly 20-100ms), we used a script from Checkout performance . But Carrier Services reports us at 500-600ms. Honestly I have no idea where the remaining 300-400ms would be coming from.

The other problem is the feedback loop: after each change we have to wait 5-6 days before we can even tell whether it helped or not - and even then we don’t really know what’s actually happening. At this point we’re close to the limit of how much faster we can make our request when we can’t see the real per-request time you measure on your end. Not long ago everything was around 400ms. It now also looks like we’ll lose the Built for Shopify badge, because there isn’t even a hint of us getting under 500ms.

screenshot 2010
screenshot 20102004×968 48.4 KB

This is exactly why per-request timing (or the Trace IDs behind the aggregate) would help so much -

right now I can’t reconcile our numbers with the dashboard and I’m essentially guessing. If there’s any way to see what’s included in that aggregate (e.g. whether it counts network/checkout overhead beyond the endpoint), that would point me in the right direction.

Thanks!

7

Hey @Jan_Ther - if it’s helpful, we’ve just managed to get under 500ms. The biggest impact was moving our servers over to US-EAST (from EU-WEST). That cut 250-300ms off that Shopify was adding in their reporting (We also tried as much as we could to leverage caching, mongoDB optimisation etc, but that alone didn’t work (hovering just above 500ms like you are currently).

8

Hey @Alex_Hooper, thanks a lot for the tips - and congrats on getting under 500ms! That’s really useful to know, and it lines up with what we’re seeing: any server-side optimization barely moves the needle in Shopify’s reporting.

The thing I’m unsure about is the region. Our clients are in Europe, and after testing latency between our server and the stores it was mostly 20-100ms. So honestly I have no idea where I’d move the server, or whether it would help or make things worse. Moving from EU to US feels counterintuitive when we have EU clients that should be served from EU servers - unless I’m wrong about how this works and the routing is handled differently.

If Shopify calls the callback from US servers rather than EU ones (even though these are EU stores), then moving to US-EAST would make sense. But how can we know they’ll always call from the US and not from the EU? If it varies, optimizing for one region could just make the other worse. A lot of open questions here @Wes-Dev-Shopify @Liam-Shopify

Out of curiosity, are your clients in the EU or the US? It feels like a bit of guesswork as to how Shopify’s servers behave and where they actually call the callback from. :smiley:

The worst part is that there’s no real way to test this. The only option is to try it straight in production, which isn’t ideal, especially when we only find out the result several days later. So it ends up being a slow, blind experiment on live traffic.

I hope you can keep it under 500; I’ll try to do the same :smiley:

9

Hey Jan - yeah I think that’s the key bit. We’re mostly UK/EU stores too so I had the same assumption as you, that EU stores would get called from the EU. It turns out all requests even in EU use Shopify US servers.

We’ve got an EU server and a US one, routed by where the caller is, and the EU one got no carrier service traffic - it all geolocated to the US, even for our UK stores. So the callback isn’t coming from where the store or the shopper is, it’s Shopify’s US side (for us anyway).

Only caveat is I can only see our side so I can’t promise it’s always US for everyone, but it was 100% for us.

If you want to check before moving, you could just log the IP the callbacks come in on and geolocate it (or stick cloudfront or similar in front and look at the viewer country). That’d tell you where yours are actually coming from without doing the whole move blind.

10

Absolutely brilliant tip, Alex - logging the callback IP and geolocating it never even crossed my mind, and it could be a real game changer. Right now it looks like the only way to actually figure this out without moving blind. If it geolocates to the US for us too (which, given your results, is pretty likely), the plan would be to spin up a separate microservice just for the callback with its own URL and switch the stores over to it.

Really interesting behaviour, by the way . especially since the docs recommend testing latency between the app and the store, when in reality the callback comes from Shopify’s US side regardless of where the store is.

Thanks again for the great tips, Alex - glad I’ve got you here. :slight_smile:

11

Hey @Jan_Ther and @Alex_Hooper - thanks for all the detail here and Alex, great tip on logging the callback IP.

Apologies for the delay in response as well. I did some more digging here and I can confirm a few things.

We time the entire outbound call - DNS lookup, TCP connection, TLS handshake, sending the request, and waiting for and reading your full response. One detail that matters for distant endpoints: a fresh connection is opened for each rate request, so the TLS handshake is paid on every call rather than reused. For a server far from the originating region, that handshake plus network transit is usually where the gap between your ~100ms server time and the reported ~500ms comes from.

On the dashboard number not adding up: the response time shown is a p95, not an average. That’s worth keeping in mind because a p95 tracks your slowest 5% of calls, so a few hundred fast single-digit-millisecond requests won’t pull it down the way an average would. If you have a few dozen calls landing in the 200-500ms range, those are what the p95 reflects.

On where the requests originate: they come from the region closest to the merchant. The important nuance is that a store’s hosting region isn’t always the same as where the merchant or the shopper is located. That’s why logging the inbound callback IP, like Alex suggested, is a solid way to see the routing for your own shops. I’d just be careful about treating it as universal, since it can vary from one store to the next rather than always being US.

On per-request timing and logs: you’re right that we don’t expose these to partners today, only the aggregate metrics in the Partner Dashboard. Per-request visibility is a valid feature request and one that’s come up a few times, so I’m going to ensure this feedback is heard alongside your notes here on why the slow feedback loop makes it hard to validate changes as a formal feature request.

Also, if it’s helpful, please open an authenticated support ticket at help.shopify.com referencing this thread and I can have the team look at your specific connect-versus-request split. That would tell us whether your gap is the connection setup or the network transit, which would hopefully give you some additional visibility in testing.

Let me know if I can clarify anything further and I hope this helps!