Sidekick app extension: tools registered correctly but Sidekick returns "Skill not found" — intermittent across chat sessions

Extensions
1

Symptom: Sidekick reports our deployed app’s skill as “not found” when trying to invoke any of our 7 tools — but the behavior is intermittent. In the same dev store, same active app version, same browser session, two back-to-back Sidekick chats can produce different outcomes: one successfully invokes our tools and returns real data, the next responds “the skill isn’t available to me in this session.”

App identification:

  • App handle: weaver
  • App ID: c64a35d9e6e89ce367704b9cf1add9e4
  • Org: 0xlab
  • Active version: weaver-13 (deployed 2026-05-21)
  • Last version where Sidekick reliably invoked our tools: weaver-7 (2026-05-20 09:07:58 UTC)
  • Dev stores tested: weaver-dev-sy57ffgo.myshopify.com and a brand-new dev store created today — same failure on both.

Extension setup (current, all required fields present and well-formed):

  • Type: ui_extension
  • Target: admin.app.tools.data
  • API version: 2026-04
  • [sidekick] extensions_summary in shopify.app.toml: present, scoped only to currently-deployed connectors (Klaviyo, Notion, Google Sheets)
  • Extension description in shopify.extension.toml: present, ~250 chars, lists explicit “Use when the merchant asks about X” trigger phrases
  • tools.json: 7 tools registered, all with non-empty inputSchema.properties
  • instructions.md: per-tool invocation guidance for Klaviyo, Notion, Sheets

Key evidence — Sidekick’s own diagnostic response:

When I asked Sidekick to invoke tools by ID directly, its response was:

“every attempt to activate the Weaver skill has returned ‘Skill not found’”

“The root issue is that Weaver’s skill isn’t registering on my end, even though the app is installed.”

So Sidekick itself is telling us its backend returns “Skill not found” for our skill ID. This explains the intermittent behavior — when Sidekick’s cache serves a “registered” state, our tools work; when it serves the “not found” state, they don’t.

What we’ve tried (no effect):

  1. Uninstalled and reinstalled the app on the dev store
  2. Created a brand-new dev store and installed from scratch — same failure
  3. Rolled back to multiple previous app versions (weaver-7, 9, 10, 11) that previously worked — same failure
  4. Multiple redeploys via shopify app deploy; each succeeds with Included './tools.json' and Included './instructions.md'
  5. Verified shopify app info shows the extension as deployed, not draft
  6. Browser DevTools console: when tools work, [weaver] calling backend logs appear; when they don’t work, no fetch is even attempted — confirms the tool isn’t registered for that chat session

Suspected root cause:

For a brief window we had a deploy with inputSchema: { type: "object", properties: {} } on one tool (empty properties object), and our extensions_summary advertised connectors (Slack, Gorgias, Meta Ads) that weren’t actually in tools.json. Both have since been fixed in multiple subsequent deploys. We suspect Shopify’s Sidekick skill registry cached our app as “broken” during that window, and the fix is propagating unevenly — explaining the intermittent recovery across sessions.

Can the Sidekick team force a re-registration / cache invalidation for app c64a35d9e6e89ce367704b9cf1add9e4? Or confirm whether there’s a known propagation delay for tool registry updates that we should just wait out?

Happy to provide more logs, request IDs, or anything else useful.

2

Hi @mhynes

I believe you’re experiencing this because sidekick app extensions are not actually publicly available yet. Although the docs are live, the extensions are only currently accessible to partners in a closed beta program.

3

Hi again @mhynes

You may be seeing these error if you’re testing on a production/ non-dev store - if you try on a dev store created from the dev dash, these skills should be registered by Sidekick.

4

Thanks Liam — we are on a dev store created via the dev dashboard (weaver-dev-sy57ffgo.myshopify.com), and Sidekick was successfully invoking our tools as recently as two days ago (2026-05-20) on that same store. So the closed-beta gating isn’t the blocker for us.

The mystery is the intermittent behavior since then. Concrete observations within the last ~24 hours, all on the same dev-dash dev store and same active app version:

Chat session A: I ask “what are my Klaviyo flows?”, Sidekick invokes klaviyo_flow_performance, returns real data — works perfectly

Chat session B (next chat, same browser tab, refreshed): same prompt, Sidekick responds “the skill isn’t available to me in this session”

When I asked Sidekick to invoke a specific tool ID, its exact words were: “every attempt to activate the Weaver skill has returned ‘Skill not found’”

So the registration appears to be there for some sessions and missing for others. We’ve also tested on a freshly created dev-dash dev store (same org) and saw the same failure — but with the same chat-to-chat intermittence.

Is there a known propagation/cache layer on the Sidekick side that could be in an inconsistent state for our app (c64a35d9e6e89ce367704b9cf1add9e4)? We had a brief window with a malformed deploy (empty inputSchema.properties on one tool) which we suspect may have poisoned the registry during that window, and the fix is now propagating unevenly. Both issues have been fixed in multiple subsequent deploys (current active: weaver-13).

5

Hey Liam, was about to look at using these myself, perhaps the docs should have a banner mentioning that they’re still closed (other pages normally do for things that aren’t publicly available).

6

Not sure about closed beta, is there an application link anywhere for it? Or is it invite only for now?

UPDATE:
So today it seems to be working again for the most part.

Is there any word on the intermittence of this? or is it just behaving this way because I am not part of the closed beta?

7

Hey Liam, is there a way to sign up for the beta? We would love to explore this for one of our apps. :folded_hands:

8

Hi @tom_digismoothie - the team running the beta are not accepting new applicants. However you should be able to test Sidekick App Extensions with apps when they’re installed on dev stores. It’s just not in General Availability currently.

9

Hey Liam, just want to confirm this because the ‘Submit your interest’ banner is appearing on Sidekick app extensions

Screenshot 2026-06-11 at 11.45.13
Screenshot 2026-06-11 at 11.45.131658×234 23.1 KB

10

Same symptom here, with numbers.

App 6981071 on gw-qa-test-77.myshopify.com, two extensions (admin.app.tools.data and admin.app.intent.render), api_version = "2026-04", CLI 4.8.0.

Across two rounds on a deployed version whose source is content-hashed (so I can prove the builds were byte-identical), Sidekick resolved the skill app-6981071 in 1 of 3 sessions in the first round and 4 of 5 in the second. A failing session returns Skill not found: app-6981071; retrying as app:app-6981071 fails identically.

Two details that may narrow it down:

  1. In a failing session the model can still read /workspace/agents/skills/apps/app-6981071/SKILL.md successfully - the generated file is present and complete, with our tools and instructions in it. So the file exists while the registry lookup misses.
  2. The first round ran within ~30 minutes of an app release; the second ran on a version that had been live ~20 hours. Small sample, but 33% → 80%. Is skill registration expected to lag behind app release, and if so, by roughly how much?

Security scan events in the dev dashboard are OK for every entry on this app, so this is not a scan block.

Happy to supply exact session timestamps if that helps.

11

Our app is Candy Rack (public, production). Its admin.app.tools.data extension has 2 read-only analytics tools. API version is 2026-04, and the active version was deployed 2026-10-08. We see the same intermittent pattern on production merchant stores:

  • Chat 1: Sidekick shows “Checked app capabilities”, then answers that Candy Rack’s data “is unavailable to me here” and asks for a screenshot or export.
  • Chat 2, right after, same tab and same question: Sidekick calls get_top_offers and answers with real data.

What we’ve verified:

  • Failing chats never reach our backend. The tool handler fetches our app backend right away. During failing chats, our server logs show no CORS preflight or request from extensions.shopifycdn.com, so the handler never ran. That matches what @mhynes seeing no fetch in DevTools.
  • Working chats run end to end. In a HAR of a working chat, Sidekick first loads the extension’s tools.json and script from cdn.shopify.com/partners-extensions-scripts-bucket/ui_extension/handle//version//… Then the preflight and the POST to our backend return 200, and Sidekick uses the data.
  • It also works for real merchants. Tool calls from other production merchant stores succeed when Sidekick does call them. So auth, CORS and our backend are fine. Whether a chat gets the tools seems to come down to the session.