We’re getting a whole bunch of these errors as of the last 5 minutes.
Good morning, @Alan_G, @Liam-Shopify, I think this is a widespread issue. Are you seeing anything on your end? We’re getting these errors on multiple API endpoints, from several different merchants.
Hi @flavio-b
Can you give more context? Are these failing requests for the Admin API? Storefront API?
Any common resources or payloads between these failing requests?
The error message you describe sounds like a network level failure, so are you certain your system is routing outbound traffic correctly?
We are also seeing this for a large number of our merchants, starting around 10:30am ET. Seeing this over the GraphQL admin APIs.
EDIT: Looks like it’s recovering now.
Hey folks - thanks for flagging this. Glad to hear it seems to be recovering though. I’m still happy to look into this though if need be. Could you share some specific error messages if possible? I’m not seeing any other reports of this on my end as of now, but I’m happy to look into this if you’re able to share any specific domains that are affected.
Hello Alan, it seems to have recovered for us, too. I don’t have a specific error message other than Failed to open TCP connection to [domain] (execution expired), which is a network-level message from Ruby’s IO modules, I think. When the process tries to use any Shopify API (REST and GraphQL admin), it can’t even open the connection. So the problem is likely some load balancer or other infra layer. This went on for about 45 minutes. If you could confirm anything on your end, I’d appreciate it. I don’t see any error status from our provider AWS and no app changes were made this morning.
We are also on AWS and it affected multiple machines. So if this was a problem that Shopify didn’t see internal errors for, it’s very possible it was a network issue somewhere between (perhaps in AWS infrastructure).
Either way, it seems resolved now, which is good. ![]()
Thanks @flavio-b @marcbaumbach - I’m still not seeing anything definitive on our end that would explain the issue, but I’ll dig into this further internally and see if I can share any info with you. Glad things are back up and running!
Hey folks, just following up on this - if you folks were able/open to share a traceroute from your host we could take that info to investigate further on our end. We’ve been digging into this but do require a bit more info to investigate thoroughly (especially since it seems to have been mitigated). Hope this helps, feel free to ping me here if we can help out further.
Hi Alan, unfortunately I don’t have that data, just the error messages which don’t help much. If this errors happens again, I’ll let you know here.
Thanks @flavio-b , just ping me here if this pops up again and we can take a look for sure!