0

I have to orchestrate a long-running task which involves several tasks, some of the tasks are 3rd party integrations and some others are CPU intensive, so every task has its own worker process. The tasks normally needs the result of the previous long-running task to create something else and pass along the data until I finish the process.

So far I have a lot of events moving around and it looks weird.

Is there a good architectural pattern I can use to improve this?

I will give an example of the process:

  • TASK1: Generate some speech using IA in text. (It may take several seconds)
  • TASK2: Synthesize the TASK1 text and convert into voice. (It may take several minutes so I have to poll the API until its done)
  • TASK3: Transcribe the voice so I can merge into a video later. (It may take several minutes and I have to poll AWS Transcribe API until its completed)
  • TASK4: Take outputs from TASK1, TASK2 and TASK3 to render a video clip. (It take several minutes).

I was looking into Saga pattern / Process manager pattern, but all that looks more micro services oriented. What would be a proper solution to this from the orchestration/architecture standpoint?

I am using SQS queues to communicate with worker processes and every worker is a NodeJS processor running in Heroku. The whole process initiates with an API request that triggers the workers.

Hugo Marcelo Del Negro
  • 138
  • 1
  • 11
  • Saga pattern is for distributed transactions so it is not really needed here for in-process task/job synchronisation. I think you are looking for [pipeline](https://java-design-patterns.com/patterns/pipeline/) instead. – Peter Csala Apr 13 '23 at 14:50
  • @PeterCsala thanks for your answer! I like the pipeline idea, the only problem is that every task defined above are large tasks and some of them initiates 3rd party jobs that takes minutes to run and I have to poll those integrations and wait to finish before I continue with the next one. So each task is a different worker. Now every worker calls the next one via a SQS message. Somehow I'd need to orchestrate them right?. – Hugo Marcelo Del Negro Apr 13 '23 at 15:08
  • SQS was not even mentioned in the question. If SQS is involved then it is not an in-process synchronisation problem. Could you please update your question to describe the current solution in more details? – Peter Csala Apr 13 '23 at 15:32
  • @PeterCsala sure! Its not an in-process thing, every worker is specialized in something, even one of them render videos so it requires special resources requirements. – Hugo Marcelo Del Negro Apr 13 '23 at 15:55

0 Answers0