0

I'm using Google Cloud / Google Container Engine, and I have infrastructure scripts where I create a cluster and node pools and then perform operations on the node pools once they're setup.

In my scripts, I need to make sure that the node pools are setup and the cluster is in a ready state before proceeding.

It appears that the gcloud container node-pools create command has no --wait or --no-async option, unlike the very similar gcloud container node-pools delete. Is there such an option for create? Else, is there a recommended way to "wait" until the node pool is ready?

I believe in my bash scripts, after creating a node pool, I can do a while loop, periodically checking the value of e.g. gcloud container clusters describe myclustername --zone myzone | tail -n 2 | grep "status" | awk '{print $2}' until I get "RUNNING" back, but perhaps there is a more elegant approach?

(It would be nice to have parity in options for creating and deleting node pools!)

davewy
  • 165
  • 1
  • 9
  • I cannot think on a much different workaround. Could you provide some additional details on your use case? What kind of operations do you need to run on the node pools after they get ready? – Carlos Jan 30 '17 at 23:48
  • @davewy, please feel free to submit a feature request on [google-cloud-sdk issue tracker](https://code.google.com/p/google-cloud-sdk/issues/list) for adding this flags to `gcloud container node-pools create` command. – Kamran Mar 02 '17 at 04:24

2 Answers2

2

As of this writing the gcloud container node-pools create command is sync by default but doesn't have an --async or --no-wait option. This isn't so bad from a shell scripting perspective as it's easy enough to background a command, and would solve your particular issue.

An alternative to deal with the original behavior would have been to use --log-http to grab the operation ID and feed it to gcloud container operations wait (which is a bit messy and requires scraping the output). This suggests another feature which would be nice to have, which is for async commands to echo the operation ID.

Adam
  • 868
  • 5
  • 12
0

I've created this script that will wait for any container operations that are not done yet.

wait_container_operations.sh

#!/bin/bash
# This scripts runs gcloud container `operations describe` and `wait` for all found operations with `list --filter=STATUS!=DONE`
# API outlined here https://cloud.google.com/sdk/gcloud/reference/compute/operations/
set -euo pipefail
IFS=$'\n\t'

source_dir="$(dirname "$0")";
current_dir="$(pwd)";
echo "Listing, describing and awaiting NOT-DONE container-operations";

function sourceClusterZone(){
    cd $source_dir;
    source ./cluster_zone.sh;
    cd $current_dir;
}

queryNotDone(){ gcloud container operations list --filter=STATUS!=DONE --sort-by='~START_TIME'; }

listNotDone(){ queryNotDone | awk '{if (NR!=1) {print $1;}}'; }

sleep 2;
LISTNOTDONE=(`listNotDone`);
echo "\""${LISTNOTDONE[@]}"\"";
if (( ${#LISTNOTDONE[@]} )); then
sourceClusterZone;
for notDone in ${LISTNOTDONE[@]}
do
    echo "Waiting for $notDone";
    gcloud container operations describe $notDone --zone="${ZONE}";
    gcloud container operations wait $notDone --zone="${ZONE}";
    echo "Done with $notDone";
done
else
    echo 'Not Waiting';
fi

cluster_zone.sh (in same dir)

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

kubeClusterOptions(){ kubectl config current-context | awk -F '_' 'BEGIN { ORS=" " }; {print $4} {print $3}'; }

declare -a OPTIONS;
IFS=' ' read -a OPTIONS <<< `kubeClusterOptions`; IFS=$'\n\t';

while getopts c:z: option 
    do 
        case "${option}" 
        in 
        c) CLUSTER=${OPTARG:-${OPTIONS[0]}};; 
        z) ZONE=${OPTARG:-${OPTIONS[1]}};; 
    esac 
done
export CLUSTER=${CLUSTER:-${OPTIONS[0]}};
export ZONE=${ZONE:-${OPTIONS[1]}};

You can configure the scripts with custom cluster and zone with -c YOUR_CLUSTER and -z YOUR_ZONE. It will take the configuration from kubectl config current-context if you don't specify any.

Nico
  • 101
  • 1