How to calculate GraphQL cost estimates

GraphQL Admin API Troubleshooting
1

Subject

Shopify’s GraphQL cost requestedQueryCost and cost actualQueryCost.

Speculation

I feel quite confident that Shopify, being a ruby company, is using https://graphql-ruby.org/ as their GraphQL library.

Additionally, I’m quite confident that they are using a highly modified version of GraphQL::Analysis::QueryComplexity to calculate the cost of GraphQL queries.

New work

I’ve started some work to replicate their algorithm: graphql-ruby/lib/graphql/analysis/shopify_complexity.rb at shopify · hrdwdmrbl/graphql-ruby · GitHub but it isn’t yet accurate.

Request for contributions

If anyone has any specific technical insight into what Shopify might be doing or how they might be calculating costs, I would love to hear it!

References

  1. They wrote a blog post a while back Rate Limiting GraphQL APIs by Calculating Query Complexity - Shopify but I think that their basic documentation is more useful, even though it isn’t much: Shopify API limits
2

Good intuitions. The specific cost formulas are intentionally closed to protect our ability to make necessary adjustments. I can offer a few general insights though.

  • If you haven’t seen it, we wrote the GraphQL Ruby complexity cost explanation that frames a very tangible summary of the Shopify costing structure.
  • For simplicity, that summary describes all linear tally strategies. We do incorporate logarithmic scaling into connection fields to cost them more favorably for a client.
  • Per the docs, the cost of select fields may be adjusted. This is particularly true of mutation fields where costs may scale by input size.
  • Estimated costs are a static calculation (idempotent to input). Changing these costs is considered breaking and will only ever be done at a version cutover.
  • Actual costs are a dynamic calculation based on response size that informs a partial throttle refund. We reserve the right to tune these as necessary while maintaining a generally consistent costing basis.
1 Like
3

Thank you for those references!

  • I should have specified that, at present I am only interested in:
    • Queries - No mutations.
    • Cost estimate vs actual costs - An estimate is all a client can hope to do
    • Manual field costs - Mostly ignore them. They seem finicky.
  • Re: Logarithmic scaling - This is FANTASTIC to know about. I had already seen hints of this, but the confirmation will help the effort immensely!

–

With respect, you say both:

  • “The specific cost formulas are intentionally closed to protect our ability to make necessary adjustments”
  • “Estimated costs are a static calculation (idempotent to input). Changing these costs is considered breaking and will only ever be done at a version cutover.”

To me this means that you could release the estimate formula for queries without constraining yourselves any further than you already are.

  • Cost estimates could be versioned - No promise about preserving costs between versions
  • No support for actual cost - Impossible for clients anyway
4

It is subtle in the docs, but you can get a detailed breakdown of how each field contributes to the actual cost by including the header Shopify-GraphQL-Cost-Debug=1 in your request.

Rate limits influenced by complexity is a recurring friction point and something we don’t think we’ve solved very well. The math is not a big secret but opaque due to change as we continue to invest and tune its behaviour.

1 Like
5

When using Shopify-GraphQL-Cost-Debug,in the fields response, does definedCost always mean that Shopify has set a fixed cost for something?

For example, looking at this query:

inventoryItem(id: $inventoryItemId) {
  id
  locationsCount {
    count
  }
  measurement {
    id
    weight {
      value
      unit
    }
  }
  trackedEditable {
    locked
  }
  unitCost {
    amount
    currencyCode
  }
}

I would have assumed a cost of 6. I count up the nested objects: inventoryItem, locationsCount, measurement, weight, trackedEditable, unitCost.

But I think that I might have it backwards. Instead I should assume all nested objects are 0, except for those that Shopify has a definedCost for. The fields says definedCost = 1 for: locationsCount, trackedEditable, inventoryItem

6

That is a reasonable default assumption and matches documentation. There are instances where we’ve overridden the default, generally to a lower cost.

Objects that are more analogous to primitive value objects (e.g. unitCost, measurement) may be assigned 0 cost. The thinking being they are an information wrapper, not an object entity with an identity or lifecycle.

Fields that return a Count object have a default cost of 10 as it can be more intensive to compute, but locationsCount applies an override to 1.

1 Like
7

So it sounds like my next guess

But I think that I might have it backwards. Instead I should assume all nested objects are 0, except for those that Shopify has a definedCost for.

is just wrong.

As you said:

Objects that are more analogous to primitive value objects (e.g. unitCost, measurement) may be assigned 0 cost.

I should assume that all nested fields cost 1, EXCEPT for specific zero-cost fields.

8

Alright, well, my implementation ends up being pretty simple: graphql-ruby/lib/graphql/analysis/shopify_complexity.rb at shopify · hrdwdmrbl/graphql-ruby · GitHub

Highlights

Results

  • It works exactly right most of the time
  • It is biased towards over-estimating the cost because it assumes that all objects are cost 1.
9

At the end of the day, the best way to cost a query is to run it and get the field-level itemized costs. That’ll tell you what each field cost. We do ultimately want customized field costs to be documented, although there’s no set timeline on when or how that will get surfaced.

For connections, you might as well use the real math rather than trying to reverse engineer it – it’s not secret, although again we don’t document it because we reserve the right to change it in a non-breaking manner. At present, each connection field is its own cost envelope computed as:

cost = 2
cost += children_cost * (2 * Math.log([2, sizing].max)).floor if sizing > 0
1 Like
10

the best way to cost a query is to run it and get the field-level itemized costs

True, but for software there are always many variables at play.

I should have explained my motivation: A concurrent system with maximum efficiency.

Setup

My application is very data-heavy.

When multiple tasks are going on at the same time for the same store, my system has to share resources (points). Right now we use a lock so that only 1 can request at a time, AND after each query the caller records the returned points. Then the next caller can look at the number of available points and the time since the last call to determine whether it can expect its call to succeed. However right now, clients do not know the cost of the query that they are about to make. Therefore clients always assume that their query will use the maximum number of points: 1000 points. So the client will wait until at least 1000 points are available.

Limitation

That system is already quite efficient. I believe that it is as efficient as possible _when the system as a whole needs to use more points than the maximumAvailable.

However, because it is pessimistic WRT the points usage of every call, it is less efficient when a set of queries needs to use more than the maximumAvailable.

If the system needs to use fewer than the maximum available, then clients could all request in parallel.

Solution

If a client could know the cost of its own query ahead of time, then it could know whether it can safely make its query without waiting for the lock.

The first query would know (pessimistic estimate) its own query cost and deduct that from the number available. The second query would now (pessimistic estimate) the cost of its query too, and could immediately decide whether to run without waiting for the response from Shopify about the cost of the first query.

Knowing the cost of its own query would also allow clients to make queries in parallel without worry that one (or more) might fail due to a race condition.

Workaround

So yes, right now I could exclusively use queries that always have the same cost and meticulously record the cost of every query (and keep it up to date) so that each query would know its cost ahead of time. But a better solution would be an automated system that can estimate (accurately or at least pessimistically) the cost of any query ahead of time.

I may release my code publicly so that others can benefit from a maximally efficient and distributed query system.

1 Like
11

Alright, made the updates based on information from @gmac . Many thanks.

Final implementation: GitHub - hrdwdmrbl/graphql-ruby: A Shopify GraphQL Cost estimator