0

Reading this article: https://aws.amazon.com/blogs/containers/running-airflow-on-aws-fargate/

It isn't clear to me; is it possible to run MWAA and execute jobs on ECS Fargate? Or to execute jobs on Ecstatics Fargate do you need to run then entire Airflow stack on ECS Fargate?

Myles McDonnell
  • 12,943
  • 17
  • 66
  • 116

2 Answers2

2

I'd recommend reading this document on AWS MWAAs, specifically the section on Architecture as it should provide you with more context.

It isn't clear to me; is it possible to run MWAA and execute jobs on ECS Fargate?

Yes. A MWAA runs it's Airflow components (scheduler, worker etc) on Fargate and will automatically execute it's jobs in Fargate containers. It will also scale the number of containers to meet demand.

There are also a plethora of Airflow integrations out there that you can use to offload the tasks/nodes within a DAG to other services (such as ECS, Batch etc.)

Josh
  • 1,556
  • 1
  • 10
  • 21
1

It is not well documented but it is possible. We are successfully running MWAA with ECS task operator and custom images.

Basically you'll need the following:

  • MWAA environment
  • MWAA execution role with added permissions to run tasks in ECS and access CloudWatch logs
  • ECS Tasks definitions

You'll also need to add apache-airflow[amazon] in MWAA Requirements file.

Jacek Sztandera
  • 904
  • 6
  • 5