Custom Specification Filtering

Custom Data
,
1

Hello Shopify Dev Team.

Question?
Can you filter through a collection list that has over 3000 products?

Context:
I am building a shopify application that is responsible for importing data from my company API. My company deals with large datasets such as importing over 5-10 thousand products to shopify. Each product can have up to 20 specification descriptions/values pairs. This is the problem. How can shopfiy, hold and store this information and make it filterable at a collection level?

Attempted Solution FAILED (1):

First solution that failed was creating a metafield definition by admin graphql call that had a type: “json”, then setting that metafield by injecting it into my bulkoperation product create where the metafield value: was a json string of current product specification description/values. Here is an example result for one product being sent into the bulkOperation productCreate:

{"input":{"title":"00024-2862-401","descriptionHtml":"A hex cap screw is a cap screw with a hexagonal head, designed to be driven by a wrench (spanner). An ASME B18.2.1 compliant cap screw has somewhat tighter tolerances than a hex bolt for the head height and the shank length. The nature of the tolerance difference allows an ASME B18.2.1 hex cap screw to always fit where a hex bolt is installed but a hex bolt could be slightly too large to be used where a hex cap screw is designed in.","productType":"006","handle":"00024-2862-401-bolts","metafields":[{"namespace":"inxsql_product","key":"inxsql_specs","type":"json","value":"{\"speclist\":[{\"specValue\":\"1/2\",\"specDesc\":\"Diameter\"},{\"specValue\":\"13\",\"specDesc\":\"Thread Pitch\"},{\"specValue\":\"7\",\"specDesc\":\"Length\"},{\"specValue\":\"Grade 2\",\"specDesc\":\"Material\"},{\"specValue\":\"Zinc Plated\",\"specDesc\":\"Finish\"}]}"}]}}

As you can see by the example I was able to store the spec description/values in a json string. Next was to access and parse this data on the frontend also know as handling the data on the store-front. The shopify liquid templating system is where the problems arise for this solution. Just a reminder I want to create custom filtering in a collection that has over 3000 products. The custom filters are based on the the specification Description with each Description holding a list of values that share that same description. The custom filter has to access all 3000 products in that collection to build out the proper custom filtering. To my understanding shopify only allows 50 items to render per page if we are looping through collection.product. This is problem. I want to access every product in the collection before the page loads so I can build out the custom filtering on specification. Because of this shopify limitation this is NOT a solution. I also tried to make an app proxy call to my shopify application to grab the products assoicated with the collection but this cause to much overhead in loading times being up to 30 seconds long. That wont work either.

ATTEMPTED SOLUTION FAILED (2)

Since the dataset for product specifications is so large per product I’d figure to create a metaobject definition and create metaobjects and link those objects back to the correct product. I attempted this successfully by doing two different method to creating a metaobject definition. First was to create a definition that has a type: single_line_text_field. Then when creating a metaobject I would take every unique specification description and map that description as the key and the value was the specification value. Here is an example:
{"input":{"id":"gid://shopify/Product/9703588692248","metafields":[{"namespace":"inxsql_product","key":"product_diameter","type":"list.metaobject_reference","value":"[\"gid://shopify/Metaobject/105676243224\"]"}]}}

As you can see Now every product would have a range of metaobjects linked to them depending on how many spec description/values they had. Now moving on to the store front I downloaded shopify’s search and discovery app. I then attempted to create 80 + filters based on the metaobject Definitions. Shopify also has limitations on how many filters can be available per store front which is 25. This means that creating metaobjects to hold spec values/description would not work either.

The second attempt for creating metaobjectDefinitions was to create only One metaobject Definition that held key, value pairs. There was two field definitions, that looked like this:
fieldDefinitions: [ { name: "Item Specification Description", key: "desc", type: "single_line_text_field", required: true, }, { name: "Item Specification Value", key: "value", type: "single_line_text_field", required: true, }, ],
I then created metaobjects and plugged in the descriptions/values and linked that object to the correct product. Now the problem lied on shopifly’s search and discovery app. Even though I had one source, I then had to group spec values and type out a group name that was the spec description even though each spec value had a description value as defined in the fieldDefinition. Shopify search and discovery app also has limitations to reading only one value from a fieldset so this means I could only display the value of either description of spec value not both. So this solution did not work.

CONCLUSION.

I need to be able to create custom filters at a collection level that can filter through the whole collection not the paginated page. Is there any other solutions that could work?

1 Like
2

Hey @Jordan_Khalil

Thanks for sharing those limitations you’re running in to. Can you share a little bit more about what the final result will look like?
Based on your explanation here, I’m seeing that on a collection page, you need the ability to further filter the products up to 20 different attributes? Could you create these as product Variants instead of storing them as metafields?

3

Hey Jordan,

Just checking in here. If you can share a little bit more about what the final result you’re looking for looks like I’d be happy to explore some other solutions! I’m just trying to envision what I could set up to test this with you.

4

Hey Kyle, I apologize for not replying. This issue was transformed and revised with a solution that no longer places spec values into a json string to be parsed on the theme. No further discussion is needed.

For the sake of transparency and documentation, here is my solution. I decided to create metafieldDefinitions based on each unique spec description value (“iwcs_spec${i}desc”) found then set the metafield by running a bulkOperation Run Mutation on product Create inserting the corresponding spec values into the correct metafield Definition by matching values. Here is an incomplete context of code where Im creating the jsonl file to be inserted into the stageduploadPath to perform the product create bulk operation mutation:

    case "shopifyproductcreate.jsonl":
      const mapHandles = new Set<string>();

      jsonlData = data
        .map((item: any) => {
          const specsArray: Array<{ specValue: string; specDesc: string }> = [];
          for (let i = 1; i <= 20; i++) {
            const specValueKey = `iwis_spec${i}value`;
            const specDescKey = `iwcs_spec${i}desc`;
            const specValue = item[specValueKey];
            const specDesc = item[specDescKey];
            if (specValue !== undefined && specDesc !== undefined) {
              specsArray.push({ specValue, specDesc });
            }
          }

          // Construct your base product object
          const productInput: any = {
            title: item.iitm_item,
            descriptionHtml: item.iweb_adddesc || item.iweb_description,
            productType: item.iwct_id
              ? item.iwct_id.toString().padStart(3, "0")
              : "000",
            handle: Sanitization.sanitizeShopifyHandle(
              item.iitm_item,
              item.iwis_webcat,
              item.iwct_parentcategory,
              mapHandles,
            ),
          };

          // If we found specs, create mutliple metafield objects-one for each spec
          //?key: `product_${Sanitization.sanitizeShopifyString(specDesc)}`,
          if (specsArray.length > 0) {
            productInput.metafields = specsArray.map(
              ({ specDesc, specValue }) => ({
                namespace: "inxsql_product",
                key: `${Sanitization.sanitizeShopifyString(specDesc)}`,
                type: "single_line_text_field",
                value: specValue,
              }),
            );
          }
          return JSON.stringify({ input: productInput });
        })
        .join("\n");

      fs.writeFileSync(filePath, jsonlData, "utf8");
      console.log(`JSONL file created at ${filePath}`);
      break;

The result, Now 80 metafield definitions are created and linked to the correct product. Since shopify has a limitation of only 250 metafield definitions we can get away with this solution.

5

Hey Jordan!

Thanks so much for sharing the solution you worked out! That will be really useful to the community.

Cheers :clinking_beer_mugs: