1

I have a use case where multiple python scripts will run in parallel on ec2. They read an SQS queue and perform operations such as download from S3, parse files, split, process, and update database. The scripts run every 5 min. If SQS is empty script does nothing and exits. If SQS has message, it gets processed by the script which may run for over an hour. So the run durations vary.

In auto scaling I see there is option to scale based on average cpu utilization but the randomness with which cpu is utilized makes it not a good indicator when to scale(e.g. the python script will take more cpu while doing disk io operations. But other times it takes memory but no cpu)

Is there a way to scale based on number of python processes running on the ec2 instance? So that when number of parallel invocations reach 10, the instances scale up?

Further, is there a way to specify that scale down should NOT happen for an ec2 instance if it has at least one python process running inside it(even if that one process is taking <5% cpu), because the scale down would terminate an ongoing process and it would be better if the scale down can happen after the python process completes

There is no load balancer in the picture.

Rohini
  • 55
  • 4
  • I looked at the idea of triggering lambda with SQS trigger for every message that SQS receives. Which would theoretically mean as the number of messages increase, more function invocations happen so it scales. But https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html specifies timeout for any lambda as 15min. This won't fit my use case as the python processing may go well over an hour – Rohini Aug 11 '21 at 05:50
  • What about controlling the Autoscaling from inside the instance? It seems to me you have some kind of control mechanism for the script so maybe use that to scale out when needed and when there's no longer need for an instance to run you can scale back in. This past July they added a feature to select which instance to terminate when scaling in. Maybe it helps https://aws.amazon.com/about-aws/whats-new/2021/07/amazon-ec2-auto-scaling-now-lets-you-control-which-instances-to-terminate-on-scale-in/ – Oscar De León Aug 12 '21 at 21:57

0 Answers0