Knowledge base app fails on some easy search queries

Agentic Commerce
1

I’ve been working on developing an AI MCP Agent for customer service of Shopify stores. I’ve been using the new storefront MCP server and I’ve struggled with some easy queries the LLM makes to the FAQs tool. Here is a clear example (images are in Spanish, sorry I’m native Spanish speaking):

IMG_2423
IMG_24231320×2279 123 KB

  1. Asked to the chat “Where can I contact you?”, the llm sends a tool call with the query “contact”

IMG_2421
IMG_24211320×2868 260 KB

  1. Tool responds that there is no answer for that
  2. I asked the chat a second time and now with the question “Contact Information?”
  3. The chat finally responds with the contact information

IMG_2422
IMG_24221320×2868 208 KB

The FAQ title I have for the contact information is “Contact information and location”, I think the simple question “What’s your contact information?” Should match the FAQ. Am I missing something?

I don’t know if there’s the possibility of creating tools per FAQ? Maybe that would help the model decide with which FAQ is best to answer depending on the user’s question.

1 Like
2

Hey @rafaremo,

Thanks for sharing those details. To ensure I have the full context, is the issue primarily with the Knowledge Base app response or is it more with your own agent implementation of the Storefront MCP server?

3

Hello @KyleG-Shopify,

Thank you for the reply. In my opinion the problem comes from the way the Search Shop Policies tool makes the queries for the correct response. For example:

  1. In the first image of the initial thread, there’s a tool calle to search_shop_policies_and_faqs that the LLM did with the query “contacto”.
  2. In the third image you can see there’s a FAQ called “Datos de Contacto y Ubicación”.
  3. In the first image again you can see the tool answered with “I’m sorry, I don’t have an answer for that question.”

I don’t know how does the search_shop_policies_and_faqs tool looks for the correct answer to return, but in my experience, something that works super good is if you expose a tool that returns all the FAQs titles and let the LLM decide which FAQ to specifically query. LLMs are super good understanding the context and intention of the questions asked and that would 10x the quality of the results when querying for the correct answer for a given question, even if the user decides tu use slang or a weird language to ask the questions.

4

Thanks for that fedback Rafael.

What I would recommend here is to make sure your agent tool is set up to properly pass the full context. When I test on my own, this is what I get:

15-48-1lm3z-y4qii
15-48-1lm3z-y4qii892×774 58.3 KB

You can see that the tool call is passing both the query and context parameters.

I used this guide here to set up the agent: Build a Storefront AI agent

And some tips here on adjusting the prompt to make it more personallized Test and customize your agent

You can also test with different LLM models to see if some work better than others for these kinds of questions.

5

Thank you @KyleG-Shopify ,

Is the context parameter new? I did nothing with my MCP client and now the LLM is sending the context parameter and giving better much answers :thinking:

Anyways, thank you for the help and I know AI products in Shopify will continue to get better as time goes by :slight_smile:

6

Thanks for confirming that it’s working now.

I’m not sure if that is new. After your reply I went and set up my own agent to be sure I had a complete working understanding of these new MCP’s.

Continue to share feedback as you use these tools so they can continue to get better and better!

Cheers.