0

Learning new things in node , I have a crude API that takes a http link => downloads the file => uploads it to my private storage. Currently I'm using RabbitMQ , where the API is the publisher and I have a consumer VM always running , waiting for task. Limitations being the VM needs to be always up and scaling the consumer is too complex

Probably their is a better solutions to handle long running tasks with scaling

Might be AWS Fargate , but I'm looking for non platform tied service if possible

Solaris
  • 674
  • 7
  • 22
  • 1
    You might not need a long-running task if an event-triggered serverless function could handle your uploads. For example, AWS's API Gateway could trigger an async Lambda which does your download/upload operation -- though with a maximum timeout of 15 minutes and temp storage of 512mb, this won't work for large files. If that's not a fit, you could buffer the jobs with a persisted queue (SQS, for example) and configure auto-scaling (of Fargate or anything auto-scalable) based on the SQS queue size. – Robert Nubel Apr 14 '20 at 04:26
  • Thank you for your suggestion @RobertNubel . Most of the jobs in my scenario would run longer than 15 mins , storage not being a problem as I'm using node streams. Also searching for vendor independent solution – Solaris Apr 14 '20 at 06:49
  • 2
    True vendor-independent autoscaling comes at a high complexity cost. Consider if you really need it -- is the cost of that upfront work really less than the possible work of having to switch cloud vendors down the line? If you think so, a container orchestration solution like Kubernetes or Nomad could do it and can be targeted against any infrastructure provider. – Robert Nubel Apr 14 '20 at 16:51

0 Answers0