Shopify AI agent giving incorrect informations

Agentic Commerce
1

While scoping the migration from non-expiring to expiring Shopify offline access tokens, I asked the agent for guidance on our current dependency versions. It responded with a code snippet and confident claims that turned out to be materially wrong in two ways that could cause real production risk if followed.

What it claimed

@shopify/shopify-app-express@^5.0.8 – supports the future.expiringOfflineAccessTokens flag.

const shopify = shopifyApp({
  future: { expiringOfflineAccessTokens: true },
});

…Reinstall / re-auth your app on each store so Shopify issues the new expiring offline token + refresh token.

But, The expiringOfflineAccessTokens flag does not exist in 5.0.8 at all. I confirmed this by pulling the actual published package (npm pack @shopify/shopify-app-express@5.0.8) and grepping the entire extracted source (compiled CJS, ESM, and TypeScript) for the literal string expiringOfflineAccessTokens — zero matches, in any file. The same check against @latest (8.0.0) finds it ~30 times, wired through real logic: future/flags.ts, auth-callback.ts, and dedicated middleware (ensure-offline-token-is-not-expired.ts, perform-token-exchange.ts). The feature was only added at 8.0.0 — three major versions ahead of what was claimed to already support it.

2

Thanks for reporting that @VK_dev. Ai agents do tend to hallucinate at times, so it’s great that you’re verifying it as you go.

If you’re using the assistant within shopify.dev there is the option to flag these instances. It helps the agent get better as well.

image
image482×126 4.39 KB