How do you handle long running task in node based apps?

Community Feedback
1

I am developing my first app that is embeded in Shopify admin. I am using all the best practices reccomended by Shopify, React, Polars, NodeJS. My app will be optimizing and automating products when it comes to titles, description, SEO description and alt text on images, depending on what user picks in automation.

The thing is when user runs the first product scan, I import all of his products , analyze them, give them seo and fullfilment score. Then if they want they can pick which one to optimize. Right now everything works as intended, but if user has over 1000 products this can turn into a issue since I dont have long running job handling yet. I was implementing these kind of things with Django using celery and redis and it worked pretty good. What are the best alternatives for this Shopify stack? Curious to hear your approach.

2

You didn’t mention your current hosting, but I’ll assume it’s Vercel, AWS Lambda, Google Functions or Cloudflare workers.

All of these have an execution limit of some kind. I too prefer a serverless environment for the real time user interactions. But the downside of unlimited scale is the lack of long running processes.

You can handle it in many different ways.

  1. Set up a pagination style API route that uses a database to mark progress, then you just continuously call it on a schedule

  2. Self host with something like BullMQ + Redis

  3. Use a managed service like Temporal (you still have to self host the execution layer)

I’ve tried all approaches, I’m most happy with Temporal, but it is expensive if you’re just starting out. But it does have a 100% self hosted option I believe.

3

If you want to get up and running fast, I’d suggest taking a look at Gadget:

1 Like
4

Yeah right now for this app I got no hosting yet, on my previous Django app I use railway that manages all the hostings for me, there I got services like Redis and standalone celery that runs.

I am considering the BullMQ and Redis. Any difficulties you had with those?

5

I am considering the BullMQ and Redis. Any difficulties you had with those?

It works well.

You have to manage more by hand, but if you prefer that approach it’s definitely solid.

You have to manage the redis capacity, spawning workers, etc. But you have flexible job chaining, delays, schedules, per tenant rate limiting.

1 Like