0

I am currently using Amazon MWAA as my Airflow. I want to have 2 types of workers nodes but currently MWAA doesn't support it. I want to have:

  1. High Compute Optimized CPU workers
  2. GPU workers

I want to create different queues for both the worker types and submit jobs to these workers nodes.

Is it possible to add an existing EC2 instance (say GPU instance) to MWAA? I only see Start and Stop EC2 operators available.

Does anyone have any pointers on this?

Manikanth Reddy
  • 31
  • 6
  • I solved it with EKS. I defined there different pool for cpu and gpu and use KubernetesPodOperator – ozs Dec 07 '22 at 12:43
  • @ozs can you give more pointers on this? So you migrated from MWAA to EKS and using KubernetesPodOperator? Any more pointers will be very appreciated. Thanks – Manikanth Reddy Dec 08 '22 at 05:12
  • EKS is working beside the MWAA. its a docker containers orchestrator. I'll add in the answer another solution and easier. – ozs Dec 08 '22 at 07:54

1 Answers1

0
  1. If you have an EKS then its possible to define a GPU pool and using KubernetesPodOperator you can run a docker under a gpu pool.
  2. Another solution its ECS (easier to define). you can see a good example to run gpu in Airflow in this article
ozs
  • 3,051
  • 1
  • 10
  • 19