I know how to assign a static external IP address to a Compute Engine, but can this be done with Google Cloud Composer (Airflow)? I'd imagine most companies need that functionality since they'd generally be writing back to a warehouse that is possibly behind a firewall, but I can't find any doc's on how to do this.
Asked
Active
Viewed 5,992 times
5
howMuchCheeseIsTooMuchCheese
- 1,602
- 1
- 22
- 30
-
1AFAIK you can get the endpoint from Kubernetes cluster running Cloud composer. – kaxil Jun 07 '18 at 22:48
-
@kaxil thanks, does that mean you think it's possible to assign a static IP for Cloud Composer since it runs on Kubernetes? I wasn't completely clear where the Python processes for Composer were running. – howMuchCheeseIsTooMuchCheese Jun 08 '18 at 12:33
-
Yes, looks like it. – kaxil Jun 08 '18 at 13:53
2 Answers
6
Now is possible. You will need to configure the Cloud Composer in a private VPC and expose the cluster using a Cloud NAT.
The documentation has some information on how to do this: https://cloud.google.com/composer/docs/concepts/private-ip
Yuri Olive
- 372
- 5
- 8
3
It's not possible to assign a static IP to the underlying GKE cluster in a Composer environment. The endpoint @kaxil mentioned is the Kubernetes master endpoint but not the GKE nodes.
If the intent is to let all outgoing network connections from Composer tasks use the same external IP, consider setting up a GCE VM with a static IP and adding a proxy service for all workloads running inside your Composer GKE cluster.
More information: https://groups.google.com/forum/#!topic/kubernetes-users/zNytc8GVB5s.
Feng Lu
- 691
- 5
- 6