as MIG instances in GCP keep getting scaled up and scaled down and instance gets new name every-time it scaled up and also MIG don’t have static IP. So I am trying Prometheus Pushgateway to get metrics. But after setting up everything I am getting metrics of scaled down instances also along with scaled up instance. And scaled down instance metric is in form of straight line. Now I don't want metrics of scaled down instance and also if there is any way that can automatically remove stop instances metrics only but don’t disturb running instance.
I have created .sh script which can check stopped instances and don't send metrics but it is not working. i have created crontab to check script but it is also not working.
#!/bin/bash
Get the list of active MIG instances
active_instances=$(gcloud compute instances list --filter="status=RUNNING" --format="value(name)") PUSHGATEWAY_SERVER=http://Address (which I have mentioned in actual script)
Get the list of existing metrics from the push gateway
existing_metrics=$(curl $PUSHGATEWAY_SERVER/metrics 2>/dev/null | grep "node-exporter" | awk '{print $NF}')
Loop through each active instance and scrape its metrics using Prometheus
for instance in $active_instances; do NODE_NAME=$instance
Check if the instance is part of a MIG and if it is still running
if gcloud compute instances describe $NODE_NAME --format="value(mig,status)" | grep -q "mig"; then instance_status=$(gcloud compute instances describe $NODE_NAME --format="value(status)") if [ "$instance_status" == "RUNNING" ]; then # Collect and push the metrics curl -s localhost:9100/metrics | curl --data-binary @- $PUSHGATEWAY_SERVER/metrics/job/node-exporter/instance/$NODE_NAME else # If the instance is not running, delete its metrics from the push gateway metric_to_delete=$(echo $existing_metrics | grep "$NODE_NAME") if [ -n "$metric_to_delete" ]; then curl -X DELETE $PUSHGATEWAY_SERVER/metrics/job/node-exporter/instance/$NODE_NAME fi fi fi done
