4

I have an SQS FIFO Queue in which I'm sending messages every time I want to process data. I sent batches of data 1-2 times a day and I want to process them with ECS Fargate all together because every message I process takes 10 minutes. Also every time I consume a message the ECS task closes.

So my plan is to find a way to trigger new Tasks based on SQS Available messages.

I found this article for ECS Fargate parallelization with SQS

  1. https://medium.com/nbtl/deploy-low-cost-ecs-tasks-based-on-sqs-queue-size-with-aws-cdk-8f5a47fc529d
  2. https://peritossolutions.com/aws/scaling-aws-ecs-fargate-on-application-events-with-sqs-lambda/

both refer to ECS Services that I'm not using due to the Desired Count variable which will make always a task run and this will lead to an unnecessary cost.

Any suggestions on how I can implement that?

dapo
  • 697
  • 1
  • 12
  • 22

1 Answers1

4

both refer to ECS Services that I'm not using due to the Desired Count variable which will make always a task run and this will lead to an unnecessary cost.

ECS Services is exactly what you need to use here. Desired count can be 0 when there are no messages in the queue, which will prevent any charges from being incurred. You should create an ECS service that polls the SQS queue, and configure the ECS auto-scaling to scale from 0 to N service instances based on the available message count in the queue.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • The service can't poll the queue if the service DesiredCount is 0. – Ronan Cunningham Jun 28 '21 at 21:52
  • 1
    The auto-scaling settings would trigger off the available record count. If available record count is > 0, then the auto-scaling settings would change the DesiredCount to 1 (or more, depending on how you configure it). When the service is running it will be polling the queue to pull the actual messages from the queue and process them. – Mark B Jun 28 '21 at 21:59
  • 1
    Cheers, I misunderstood the intent originally – Ronan Cunningham Jun 28 '21 at 22:01
  • You would need to be over cautious with scaling cooldown though if setting the DesiredCount to 0 but that may be fine. Given the user knows the size of the batch upfront, I'm pondering the trade-offs between a service with scaling and running N tasks. – Ronan Cunningham Jun 28 '21 at 22:23