Metafield - Metaobject reference write failures

Hey folks,

We’ve seen an uptick in error messages along the lines of:

[DiscountRefresher] [-------.myshopify.com] Collection metafield writes failed: Collection metafield writes had 1 failed batch(es): [{"field":["metafields","0","value"],"message":"Value must belong to the specified metaobject definition gid://shopify/MetaobjectDefinition/11791499353.","code":"INVALID_VALUE"}

Product metafield writes had 1 failed batch(es): [{"field":["metafields","4","value"],"message":"Value must belong to the specified metaobject definition gid://shopify/MetaobjectDefinition/20754956674.","code":"INVALID_VALUE"}

What’s interesting is that BOTH of these are writing the same metaobject reference list type, but there are two different gids.

This uptick appears to have started at 5am UTC this morning:

The previous errors are a fully separate issue from our side (which is now fixed).

Was there some sort of migration of Metaobject definition ids? The last deployment we did was on the 14th April (not this morning).

I’m unable to reproduce this issue, and it seems to be happening for only a few stores now (it was many more stores earlier), is this something that will self heal?

Thank you!

Hey Sammy! The timing here (sudden onset at 5am UTC today, no deploy on your side since April 14, affecting multiple stores then partially self-healing) does look like something changed on our end rather than in your app. We’re investigating a possible platform-side change around that time that may have affected how metaobject definition references are validated.

There is a known pattern where metaobject definition IDs can get out of sync with the metaobjects that reference them, which would produce exactly the INVALID_VALUE error you’re seeing. We can’t say for certain that’s what happened here without looking at the actual requests, though.

I’ve flagged this with the relevant team internally but to help us investigate, could you share the x-request-id response header from a few of the failing API calls? That will let us trace what’s happening on these specific requests.

Thanks @Donal-Shopify! We’ll work on getting some request-ids if that makes things simpler.

I’ve now confirmed the following:

  • Currently impacting 33 separate merchants
  • Not happening for all merchants
  • Still on-going (i.e not self-healing)

Based on how/when we write metafields, I would expect the number of impacted merchants to rise over time (it requires an addition/removal/update of discount(s)).

Ok, we’re adding in as much logging as possible now, but some additional information that we’ve discovered, for three stores (that we’ve looked at so far), we actually see a recurring pattern:

Prior to 2-3AM UTC 16th April, we saw errors like: “RECORD_NOT_FOUND” (Metaobject definition not found) when trying to ‘upsert’ an instance of the metaobject.

After 2-3AM UTC, that error seems to not happen anymore, which leads me to believe the Metaobject upsert succeeded. This then goes deeper into our flow whereby we were calling the metafieldSet.

The metaobject definition is Shopify managed (i.e, in the app.toml), so I’m not sure why the RECORD_NOT_FOUND ever happened in the first place.

The first time we see the RECORD_NOT_FOUND, Metaobject definition not found, appears to be: Mar 4, 7:28:19.807 PM UTC.

Here are some requestIds + the gids of the metaobjects from three different merchants:

[DiscountRefresher] [----.myshopify.com] Shop metafield write failed [requestId=fac52cb6-c853-4f1c-8665-319afd3f0112-1776761549]: ownerId=gid://shopify/Shop/63551045721, metaobjectIds=[gid://shopify/Metaobject/179926499417, gid://shopify/Metaobject/179926532185, gid://shopify/Metaobject/179926597721, gid://shopify/Metaobject/179926663257, gid://shopify/Metaobject/179926696025, gid://shopify/Metaobject/179926728793, gid://shopify/Metaobject/179926761561, gid://shopify/Metaobject/196624941145, gid://shopify/Metaobject/196624973913, gid://shopify/Metaobject/196625006681, gid://shopify/Metaobject/196625039449, gid://shopify/Metaobject/196625072217, gid://shopify/Metaobject/196625104985, gid://shopify/Metaobject/196625137753, gid://shopify/Metaobject/196625170521, gid://shopify/Metaobject/196625203289, gid://shopify/Metaobject/196625236057, gid://shopify/Metaobject/196625268825, gid://shopify/Metaobject/196625301593, gid://shopify/Metaobject/196625334361, gid://shopify/Metaobject/196625367129, gid://shopify/Metaobject/196625399897, gid://shopify/Metaobject/196625432665], errors=[{"field":["metafields","0","value"],"message":"Value must belong to the specified metaobject definition gid://shopify/MetaobjectDefinition/11791499353.","code":"INVALID_VALUE"}]

[DiscountRefresher] [-----.myshopify.com] Shop metafield write failed [requestId=c91e4eb5-a0c3-441b-bc83-d01b807ea51a-1776761433]: ownerId=gid://shopify/Shop/26324560, metaobjectIds=[gid://shopify/Metaobject/156461596775, gid://shopify/Metaobject/182444425319, gid://shopify/Metaobject/186106642535], errors=[{"field":["metafields","0","value"],"message":"Value must belong to the specified metaobject definition gid://shopify/MetaobjectDefinition/7819624551.","code":"INVALID_VALUE"}]

[DiscountRefresher] [------.myshopify.com] Shop metafield write failed [requestId=b097ab69-4751-4c49-997e-3623a8f5963e-1776761409]: ownerId=gid://shopify/Shop/70154354753, metaobjectIds=[gid://shopify/Metaobject/176678207553, gid://shopify/Metaobject/171864719425, gid://shopify/Metaobject/171864752193, gid://shopify/Metaobject/176678174785], errors=[{"field":["metafields","0","value"],"message":"Value must belong to the specified metaobject definition gid://shopify/MetaobjectDefinition/15134097473.","code":"INVALID_VALUE"}]

Thanks for pulling those together Sammy, and sorry for the wait on my end. I’ve passed the three request IDs, shop IDs, and the metaobject GIDs through to the metaobjects team internally along with the rejected definition IDs, so they can check what those metaobjects are currently attached to vs. what the metafieldsSet validator is expecting.

I’ll update here as soon as I hear back from them!

Thanks for the request IDs, @bkspace . The team took a look, and the important bit is that the metaobjects you’re passing appear to be the expected app-owned type for the Shop metafield you’re writing. This does not look like a straightforward payload or type mismatch from your side.

The affected metaobjects look like they may have a stale stored reference to an older metaobject definition. That lines up with an issue we’re tracking for app-managed metaobjects, where existing entries can retain a definition reference that no longer matches the current definition.

For a short-term test, could you try making a small edit that does not change the value on the affected metaobjects for one impacted shop, or recreate them if that is safer in your flow, then run the same metafieldsSet write again? If that clears the error for that shop, it gives us useful confirmation and a possible workaround while I keep this raised with the metaobjects team.