Airflow stored in the cloud?

Question

I would like to know if I can make the airflow UI accessible to all people who have a user, web page type. For this, I would have to connect it to a server, no? Which server do you recommend for this? I was looking around and some were using Amazon EC2.

Answer 1

If your goal is just making the airflow UI visible to public, there is a lot of solutions, where you can do it even in your local computer (of course it is not a good idea).

Before choosing the cloud provider and the service, you need to think about the requirements:

• in your team, do you have the skills and the time to manage the server? if no you need a managed service like GCP cloud composer or AWS MWAA.
• which executor yow want to use? KubernetesExecutor? CeleryExecutor on K8S? if yes you need a K8S service and not just a VM.
• do you have a huge loading? do you need a HA mode? what about the scalability?

After defining the requirements, you can choose between the options:

• Small server with LocalExecutor or CeleryExecutor on a VM -> AWS EC2 with a static IP and Route 53 for DNS name
• A scalable server in HA mode on a K8S cluser -> AWS EKS or google GKE
• A managed service and focusing only on the development part -> google cloud composer