0

My goal is to update composer dag variables using gcloud commands. I started with using single command to update one variable via question posted in this Import variables using json file in Google Cloud Composer.

I got the following error -

ERROR: (gcloud.composer.environments.run) kubectl returned non-zero status code. Make sure you have followed https://cloud.google.com/composer/docs/how-to/accessing/airflow-cli#private-ip to enable access to your private Cloud Composer environment from your machine.

I then followed that instructions under Running commands on a private IP environment section. I got my public IP of the cloud shell, and the gke cluster name. In the third step that mentions updating GKE Cluster, I ran the command to obtain EXISTING_AUTH_NETS using the following command -

gcloud container clusters describe cluster_name --zone us-central1-c \
    --format "flattened(masterAuthorizedNetworksConfig.cidrBlocks[])"

This returned None. So I went ahead and ran the following command -

gcloud container clusters update cluster_name \
    --enable-master-authorized-networks \
    --master-authorized-networks None,34.xxx.xxx.xxx

Running the above command I get the following errors -

ERROR: (gcloud.container.clusters.update) INVALID_ARGUMENT: (1) invalid value for "cluster.master_authorized_networks_config": invalid CIDR - "34.126.169.236" is not a valid CIDR range. Must be a match of regex [0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/[0-9]{1,2}, (for example, '10.120.0.0/14') (2) invalid value for "cluster.master_authorized_networks_config": invalid CIDR - "None" is not a valid CIDR range. Must be a match of regex [0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/[0-9]{1,2}, (for example, '10.120.0.0/14').

Can someone help me how to fix the error? I understand that IP address is wrong. Even running simple cloud composer commands such as this one, I am unable to do so

gcloud composer environments run composer-env-name list_dags --location us-central1 -- --report
VKarthik
  • 1,379
  • 2
  • 15
  • 30
  • 1
    You have specified an IP address (34.126.169.236) and not a CIDR (34.126.169.0/24). The formats are similar but represent different information. The first is a single IP address, the second is a network specification. – John Hanley Jun 18 '22 at 03:37
  • Thanks John. I have added the network specification and it seems to work but second error still remains the same i.e. invalid CIDR - "None" is not a valid CIDR range. Is there any other command I need to run to find master_authorized_networks_config? – VKarthik Jun 18 '22 at 21:40
  • 1
    why are you using None here: `--master-authorized-networks None,34.xxx.xxx.xxx` – John Hanley Jun 18 '22 at 22:25
  • Is your issue resolved? – Sakshi Gatyan Jun 19 '22 at 09:19
  • Thanks @JohnHanley. I removed that None and went ahead, it worked. Just had follow-up question, when running the commands in cloud shell, it looks like public ip is changing every session. If I were to package this as a shell script and have it executed, wouldn't there be deluge of authorized networks? My plan is to have the shell script run via CloudBuild (connected to github repo). Is there a way to bind it to a static ip and not having to run container cluster update commands every time before running composer variable updates? – VKarthik Jun 21 '22 at 01:20
  • 1
    You should post a new question as the answer is long. In summary, there are two types of authorization to consider. Location based (IP Address) and Identity based. Google Cloud makes extensive use of Identity Based Access Control (IBAC). In your case, using services that change public IP addresses will be a problem. You cannot assign a static address because the services are actually clusters of systems and not one system. Some services support things like NAT based static IP but others do not. However, this is a complex topic that needs its own question and lots of preparation to implement. – John Hanley Jun 21 '22 at 01:48
  • Thanks @JohnHanley. Appreciate your quick update and elaborate response. Will do some initial prep and post the question. For now this question can be marked done. Would you mind posting your first update as answer and I can close it down? – VKarthik Jun 21 '22 at 05:05

1 Answers1

1

You have specified an IP address (34.126.169.236) and not a CIDR (34.126.169.0/24). The formats are similar but represent different information. The first is a single IP address, the second is a network specification.

John Hanley
  • 74,467
  • 6
  • 95
  • 159