Call Admin API from within POS UI Extension

How to call Admin API from within POS UI Extension? I want to fetch the inventory levels at all locations for a particular product but the Product Search API in POS UI Extension does only allow you get inventory levels at current location and overall all locations. I tried making a fetch request to inside modal.jsx of POS UI extension but it says “fetch failed” whereas if I try to call the route from browser, it works fine. I tried reaching out to Shopify support but they also couldn’t resolve the issue.