2

We are in the development phase. So we are using AWS ECS cluster consisting of 2 EC2 instances. We are submitting tasks to ECS cluster using ECSOperator of Airflow. We are looking to scale this process. So we are going to use Celeryexecutor of Airflow which is used to concurrently submit and schedule tasks on Airflow.

So the question is, should we care about number of task submitted to ECS or irrespective of number of tasks submitted to ECS, it will service all the tasks without failure by any internal queuing mechanism?

Free Coder
  • 459
  • 4
  • 19
  • Any update? Did ECS handle task scheduling at scale? – sizzle Sep 06 '18 at 14:30
  • Hey, I've used Celery executor with ECS and didn't have to worry about how many tasks are sent to ECS by Airflow. Actually the process looking after scheduled tasks is Flower within Airflow, ECS just hosts the worker process you've set up. I've a question though, are you referring to Airflow tasks or ECS tasks (definition)? – marianogg9 Jan 27 '20 at 11:48

0 Answers0