Shortly: Sidekick is bad in the sense that the quality of its answers is too inconsistent, it requires the user to make use of good prompt engineering to have a chance to land satisfying results. Which is ok as a developer, but for regular users they wouldn’t use it well.
example 1:
I tried to integrate with a FAQ of the app. Just this is hard to get right. I can’t just feed a FAQ to sidekick because of the token limit, I need to engineer a system where I list the questions and answers in a way that sidekick will be able to use it correctly. Which means lot of experiments, maintenance, etc…
All that for a thing that I’m not even sure people would use. It’s better to just give a link to our documentation at this point and let chat gpt/claude/etc.. do the work.
I also found that sidekick tends to confidently hallucinate responses about what it finds in the provided FAQs, which lead to wrong information and it is dangerous for users. It takes a lot of time to get everything right so it wont hallucinate wrong answers.
So at this stage it’s not relevant to spend time for benefits probably close to 0, or even potentially negatives
example 2:
Then I tried to integrate with the “app actions”. I.e. ask sidekick to perform actions for us. Here again it’s not good:
- bad quality that requires prompt engineering to get it to work well, I cant afford spending time tweaking for a model that is not working well and that might break anytime
- there is context/size limit we cant pass a large json payload ,
- there is
3 seconds limit in the response,
- the model behind sidekick is clearly not smart enough to do it reliably. I wouldn’t be able to trust it enough to let the user act on the app through sidekick.
The only way to get good results is to connect it to our internal AI layer, but the 3 seconds limit is killing all possibilities to do so.
example 3:
Response can be randomly blocked by safety checks:
Overall, sidekick is not ready for everyone’s usage. You can also find a ton of comments on reddit about people saying roughly the same thing.
Given all these limits I don’t see any good use case for sidekick. If I want to add a feature I ask my self “what do the users need to do” and for any of the answers, there is always a better way to do it than sidekick, or sidekick limitations makes it impossible / too much efforts to do.
and lastly, given how often shopify changes and breaks everything suddenly without a notice, I can’t trust them on this. Developing and spending time on something that may no longer work or be relevant because of an update is too much of a gamble.
To summarize:
- the model/agent itself is not good enough
- too many limitations
- requires engineering and maintenance => time, but time is better invested for other priorities
- ROI-wise probably not worth it
- It’s a rabbit hole