`robots.default_groups` returns an older rule set than the robots.txt Shopify serves by default

Online Store and Theme Development
1

Short description of issue

The documented robots.txt.liquid template outputs a different, older set of rules than the robots.txt Shopify serves when no template exists, so a store can’t customise robots.txt without losing the current default.

Reproduction steps

  1. On a store with no templates/robots.txt.liquid, fetch /robots.txt and save it.
  2. Add templates/robots.txt.liquid containing the unmodified example from the robots.txt.liquid docs.
  3. Fetch /robots.txt again (we used shopify theme dev) and compare.

Expected: the same output, since the template only loops over robots.default_groups.

Actual: a different, older rule set (63 rules under User-agent: * against 45).

  • Only in the managed default: the agent/UCP comment header, Allow: / plus 20 more Allow rules, Allow: /account/login, and Disallow for /cart.js, /services, /sf_*, /<shop-id> and the locale-prefixed checkout/account paths.
  • Only in default_groups: Disallow: /search, /policies/, /apple-app-site-association and /services/login_with_shop, the *-remote product rules, and groups for Nutch, Ahrefs, MJ12bot and Pinterest.

Additional info

This isn’t minor drift: /search goes from crawlable to disallowed depending on whether a template exists. The template only loops over the object, so nothing a merchant writes can restore the current rules.

Our case: we needed to drop one rule, Disallow: /collections/*sort_by*. Google held 258 of our sorted collection URLs as “Indexed, though blocked by robots.txt”, unable to crawl them to read their canonicals; three weeks after removing the rule it’s 13. The loop didn’t reproduce our default, so we pasted the managed output in as plain text, minus that rule. Our robots.txt is now frozen at 12 Sep 2026 and won’t pick up Shopify’s changes.

Expected fix: default_groups returns the current rules and header, or the docs say it doesn’t.

Related: an August request wants to add one line without losing the rest; we wanted to remove one.

What type of topic is this

Bug report

2

Full lists, for anyone comparing:

In the managed default, missing from `default_groups`:

- The comment header: agent instructions, `/agents.md`, the UCP discovery and MCP endpoints, the “Checkouts are for humans” policy, and the terms and contact lines

- `Allow: /` and the 20 `Allow` rules for `/products/`, `/collections/`, `/pages/` and `/blogs/` paths containing account, orders or checkout

- `Allow: /account/login`

- `Disallow: /services`, `/sf_*`, `/cart.js`, `/`, and the locale-prefixed versions of `/cart/`, `/checkout`, `/checkouts/`, `/orders` and `/account`

In `default_groups`, absent from the managed default:

- `Disallow: /search`

- `Disallow: /policies/`, `/apple-app-site-association`, `/.well-known/shopify/monorail`, `/services/login_with_shop`

- The `*-remote` product URL rules

- Separate groups for Nutch, AhrefsBot, AhrefsSiteAudit, MJ12bot and Pinterest

So `/search` goes from crawlable to disallowed depending on whether a template exists.

3

@abosworth Thanks for reporting this issue. We’re looking into getting that Liquid drop and doc updated.