OpenAI Ads getting blocked for crawler issues

[taylor]

Short description of issue

Shopify blocking OpenAI crawlers for ads

Reproduction steps

Try to run ads with OpenAI.

Additional info

We’re attempting to run ads using OpenAI and they’re getting blocked with an inaccurate error message that suggests it’s being blocked by robots.txt though we’ve explicitly allowed.

Shopify and OpenAI docs show everything looks good for robots.txt so the block is happening sometime after that point. However, we’re weeks into this and have gotten nowhere.

Shopify support has been great with responses and following up with tech side, but we’re at a standstill. Basically we submit an ad, it looks great, and 24 hours later it’s blocked so we get errors on OpenAI side.

This is more of a platform issue, but not getting anywhere with support and doing the normal complaining on Twitter/X but don’t want to tag people like mad. Just need help getting directed to the right person so we can figure this part out.

What type of topic is this

Troubleshooting
Bug report

Upload screenshot(s) of issue

[Alan_G]

Hey @taylor - I dug into this a bit. I can’t confirm whether this is robots.txt or the separate network-protection layer though, so I might need a bit more info to investigate.

Could you share the exact landing URL and the approximate rejection time in UTC?

Also, could you confirm the live robots.txt explicitly includes User-agent: OAI-AdsBot with Allow: /, and that the landing page doesn’t redirect to another domain or protected page?

I can take a look at our logs to see if there’s anything odd on our end here for sure.

[taylor]

Thanks @Alan_G! I can tell you at least it’s not robots.txt issue. This is a bad error on OpenAI’s part and it’s definitely further along the line.

Can also confirm our live robots.txt explicitly allows both
OAI-AdsBot and OAI-SearchBot

I do have an open support back and forth and I can DM you the URLs and such to give you more explicit context.