Bulk operations failing on big catalogs

GraphQL Admin API Troubleshooting
1

Hi there,

In the last weeks, bulk operations for creating all products catalogs are failing for stores with huge catalogs. The main problem is:

  • They are failing at the end of the generation, after hours. I know it is at the end because I can check the number of products processed every minute.
  • Even more problematic: at least, the bulk operation failure response is providing an url or partialDataUrl, but the file there is empty.

Please could you take a look? Here are some recen bulk operation IDs:

id gid://shopify/BulkOperation/6493682794720, status FAILED, error code INTERNAL_SERVER_ERROR

id gid://shopify/BulkOperation/6490762248416, status FAILED, error code INTERNAL_SERVER_ERROR

1 Like
2

Hey @sinuhe, lot’f of bulk operations issues lately. Thanks for sharing those operation id’s.

To help narrow down where we look first, can you share a little more context on the operations you’re making here? This will also help other community members that will find this thread in the future.

Can you share:

  1. The GraphQL query you’re using for the bulk operation - This helps us check if it’s supported for bulk operations or if there are known issues with specific field combinations. Do any of the operation restrictions apply?

  2. Whether the same query works when paginating normally (non-bulk) - Testing this on one of the failing stores would immediately tell us if it’s the query structure itself or specifically the bulk operations infrastructure.

I want to add as well, if you suspect this is specific to certain store configurations rather than a broader bulk operations issue, reaching out to our merchant support team might be more helpful. They can take a closer look at the specific store setup and any configuration that might be affecting bulk operations.

3

Hi,

mutation {
  bulkOperationRunQuery(
   groupObjects: false, query: """
    {
      products (query: "published_status:published") {
        edges {
          node {
            id
            handle
            productType
            title
            description
            vendor
            publishedAt
            updatedAt
            tags
            featuredMedia {
                preview {
                    image {
                      url
                      altText
                    }
                }
            }
            variants {
              edges {
                node {
                  id
                  title
                  availableForSale
                  barcode
                  price
                  compareAtPrice
                  sku
                  image {
                    url
                    altText
                  }
                }
              }
            }
            collections(query: "collection_type:custom") {
              edges {
                node {
                  id
                  title
                }
              }
            }
          }
        }
      }
    }
    """
  ) {
    bulkOperation {
      id
      status
      errorCode
      url
      partialDataUrl
      createdAt
      completedAt
    }
    userErrors {
      field
      message
    }
  }
}

It is difficult to test. This is failing only for two stores: one with 385K products, and another with 825K products, so do it by paginating would take a long time.

The problem here is not only the failure, we know failures are possible, but also the empty partialDataUrl file.

4

Thanks for sharing that. I can see how paginating isn’t easy with that many products.

Have you done any other troubleshooting? For example, removing some of the connections to narrow down if it’s a specific to any of the requested fields? (ie removing variants, collections or featureMedia to see if it works without either of those).

5

Hi @KyleG-Shopify ,

No, we are using the smallest query we can for our purposes. Reducing fields would be useless for our use case.

For small catalogs, I would not have any problem to perform the tests you suggest. But for these huge catalogs, those tests would take hours.

I understand you need tests to find the bug… but in the end this is something that used to work and is failing now. And furthermore the error does not meet the documentation, as the partialUrl file is empty.

Please could you take a look?

6

Thanks for sharing that. In cases like this where it’s something that was working previously and is no longer, where the API was used correctly, and that can’t be replicated on a test shop, it’s usually something specific to a resource on the specific shops.

You can do some of those tests I mentioned to help you narrow down the troublesome resource. If that is not feasible, your next best option is to reach out to our support team directly through the help center.

For context, looking in to and discussing a specific resources on a shop does require proper authentication which we don’t have here in the public community.