MCP server and KnowledgeBase App

Agentic Commerce
1

I have added few FAQs to my dev store using Shopify KnowledgaBase App. Now when I try the MCP tool search_shop_policies_and_faqs, its not able to get the answer. Although in the UI of KnowledgaBase App, its shown that my question matched with 3 faqs. What might be going wrong? Is it caching issue? I have waited for 12 hours. Should I wait more?

2

Hi @Angela_Ranjeet

Just to confirm, you’ve set up the FAQs as is described here: https://help.shopify.com/en/manual/promoting-marketing/knowledge-base/managing-faqs

3

Thanks for the response @Liam-Shopify. That’s right, I have followed those instructions. When I call the tool (From UI or using curl), I don’t get a response:

curl -sX POST “https://store-url/api/mcp” \

-H ‘content-type: application/json’ -H “Cache-Control: no-cache, no-store” -H “Pragma: no-cache” \

-d '{

"jsonrpc":"2.0",

"id":3,

"method":"tools/call",

"params":{

  "name":"search_shop_policies_and_faqs",

  "arguments":{

    "query":"what is your return policy?",

    "context":"customer is asking for return policy"

  }

}

}’

{“jsonrpc”:“2.0”,“id”:3,“result”:{“content”:[{“type”:“text”,“text”:“Unable to find the answer in the knowledge base, try again”}],“isError”:true}}%

But in the UI, it shows that my question matched with 3 faqs:

image
image1224×284 42.1 KB

4

@Liam-Shopify , any updates here? I am still facing this issue.

5

Hi @Angela_Ranjeet

Still digging into this on my side - will post an update here asap

1 Like
6

Hi again @Angela_Ranjeet

This does appear to be a bug on our side - we’re working to address this as soon as possible.

7

Thanks @Liam-Shopify ! We did a bit of testing at our side and looks like as soon as the number of faqs increase beyond a threshold (around 40 +/- 5), we get into this situation. Would wait for the resolution.

8

Thanks for sharing that extra info @Angela_Ranjeet - we were able to replicate this issue so that’s helpful. Will post here when I’ve an update to share on this.

1 Like
9

Hi again @Angela_Ranjeet

Just wanted to reconnect to let you know we’re working on a fix to improve this experience. Will update here once that has shipped.

1 Like
10

@Liam-Shopify , it it fixed now?

11

We did ship some recent improvements to the tools that connect with FAQs - are you still seeing an issue on your side? If you are could you share your store (or I can DM you to gather info privately).

12

HI @Liam-Shopify , there is still some fragility in this tool. After adding a question and using curl to invoke the tool, I get 2 different answers (Success and Failure) for the same question in 2 consecutive calls:

curl -sX POST “https://<my_store>.myshopify.com/api/mcp?timestamp=$(date +%s)” \

-H ‘content-type: application/json’ -H “Cache-Control: no-cache, no-store” -H “Pragma: no-cache” \

-d '{"jsonrpc":"2.0",

"id":3,

"method":"tools/call",

"params":{

  "name":"search_shop_policies_and_faqs",

  "arguments":{

    "query":"What is your return policy?"  

  }

}
}’

{“jsonrpc”:“2.0”,“id”:3,“result”:{“content”:[{“type”:“text”,“text”:“[{\“question\”:\“What is your return policy?\”,\“answer\”:\“You can return most new, unused items within 30 days of delivery for a full refund. Clearance, engraved jewellery, and worn snow gear are final sale. \”},{\“question\”:\“What is your return window?\”,\“answer\”:\“You can return most new, unused items within 30 days of delivery for a full refund. Clearance, engraved jewellery, and worn snow gear are final sale. \”}]”}],“isError”:false}}%

curl -sX POST “https://<my_store>.myshopify.com/api/mcp?timestamp=$(date +%s)” \

-H ‘content-type: application/json’ -H “Cache-Control: no-cache, no-store” -H “Pragma: no-cache” \

-d '{ "jsonrpc":"2.0",

"id":3,

"method":"tools/call",

"params":{

  "name":"search_shop_policies_and_faqs",

  "arguments":{

    "query":"What is your return policy?"

  }

}
}’

{“jsonrpc”:“2.0”,“id”:3,“result”:{“content”:[{“type”:“text”,“text”:“Unable to find the answer in the knowledge base, try again”}],“isError”:true}}%

This is a dev store, where I am testing. Yes, we can connect and I can give you the required information.