Run a python script from the startup script

Question

An instance is created when a new file is uploaded to the storage. The startup runs a python script that generates a pdf file for the new file, uploads the pdf back to the storage and deletes the instance. Since the python script has pretty lengthy, I have stored the startup script and the python script in the same location (Cloud Storage). I have passed the paths are metadata while creating the instance. The input to the python script is the file name of the new file. I checked the logs of the instance, its throwing some errors there. Can someone point out what is it that I am doing wrong.

Edited

Error Message:

{
  "cpuPlatform": "Intel Haswell",
  "creationTimestamp": "2021-08-02T06:40:36.346-07:00",
  "deletionProtection": false,
  "disks": [
    {
      "autoDelete": true,
      "boot": true,
      "deviceName": "xyz",
      "diskSizeGb": "10",
      "guestOsFeatures": [
        {
          "type": "UEFI_COMPATIBLE"
        },
        {
          "type": "VIRTIO_SCSI_MULTIQUEUE"
        }
      ],
      "index": 0,
      "interface": "SCSI",
      "kind": "compute#attachedDisk",
      "licenses": [
        "projects/debian-cloud/global/licenses/debian-10-buster"
      ],
      "mode": "READ_WRITE",
      "source": "projects/patch-us/zones/us-central1-a/disks/instance-name",
      "type": "PERSISTENT"
    }
  ],
  "fingerprint": "XlZ7biyVpAI=",
  "id": "3984870299667155772",
  "kind": "compute#instance",
  "labelFingerprint": "42WmSpB8rSM=",
  "lastStartTimestamp": "2021-08-02T06:40:46.210-07:00",
  "machineType": "projects/project-name/zones/us-central1-a/machineTypes/e2-medium",
  "metadata": {
    "fingerprint": "f5o3Pxed5VY=",
    "items": [
      {
        "key": "startup-script-url",
        "value": "https://storage.cloud.google.com/project-name.appspot.com/start-up-script/start-script.sh"
      },
      {
        "key": "file_name",
        "value": "123456"
      },
      {
        "key": "python_script_name",
        "value": "https://storage.cloud.google.com/project-name.appspot.com/start-up-script/generate_fd_report.py"
      }
    ],
    "kind": "compute#metadata"
  },
  "name": "instance-name",
  "networkInterfaces": [
    {
      "accessConfigs": [
        {
          "kind": "compute#accessConfig",
          "name": "External NAT",
          "natIP": "35.202.255.222",
          "networkTier": "PREMIUM",
          "type": "ONE_TO_ONE_NAT"
        }
      ],
      "fingerprint": "565TD6a2Y2c=",
      "kind": "compute#networkInterface",
      "name": "nic0",
      "network": "projects/project-name/global/networks/default",
      "networkIP": "10.128.0.29",
      "stackType": "IPV4_ONLY",
      "subnetwork": "projects/project-name/regions/us-central1/subnetworks/default"
    }
  ],
  "scheduling": {
    "automaticRestart": true,
    "onHostMaintenance": "MIGRATE",
    "preemptible": false
  },
  "selfLink": "projects/project-name/zones/us-central1-a/instances/instance-name",
  "serviceAccounts": [
    {
      "email": "project-id-compute@developer.gserviceaccount.com",
      "scopes": [
        "https://www.googleapis.com/auth/cloud-platform"
      ]
    }
  ],
  "shieldedInstanceConfig": {
    "enableIntegrityMonitoring": true,
    "enableSecureBoot": false,
    "enableVtpm": true
  },
  "shieldedInstanceIntegrityPolicy": {
    "updateAutoLearnPolicy": true
  },
  "startRestricted": false,
  "status": "RUNNING",
  "tags": {
    "fingerprint": "42WmSpB8rSM="
  },
  "zone": "projects/project-name/zones/us-central1-a"
}

start-script-sh

#! /bin/bash
ECG_FILE_PATH = $(curl http://metadata/computeMetadata/v1/instance/attributes/file_path -H "Metadata-Flavor: Google")
PYTHON_FILE_PATH = $(curl http://metadata/computeMetadata/v1/instance/attributes/python_script_name -H "Metadata-Flavor: Google")
ECG_FILE_NAME = $(curl http://metadata/computeMetadata/v1/instance/attributes/file_name -H "Metadata-Flavor: Google")

curl -s -o generate_fd.py PYTHON_FILE_PATH

chmod +x generate_fd.py
python3 generate_fd.py ECG_FILE_PATH &

generate_fd_report.py

#!/usr/bin/env python3
def main(file_name):
    print("Hello")


main(file_name)

Logs

I have uploaded both the startup script and the generate_fd_report.py to the Cloud Storage. And the path of both these files is added to the meta data, while creating the VM instance. Does this answer your question @jabbson — Jasmine, Aug 02 '21 at 14:47
This is fine, to execute these files on the host you need to download them first, where do you do that? — jabbson, Aug 02 '21 at 14:48
I think I am not doing that. I assumed since the startup script and the python script in the folder, it will execute. — Jasmine, Aug 02 '21 at 14:50
The curl to the metadata server from the startup script gets the `path to the file` as specified in metadata field during the vm creation, then you need to also download it, before you can execute it. — jabbson, Aug 02 '21 at 14:52

score 1 · Accepted Answer · answered Aug 02 '21 at 15:04

1

To download the script, URL path to which you've saved as the metadata value is as follows:

curl -s -o filename.txt $(curl -s http://metadata/computeMetadata/v1/instance/attributes/filename -H "Metadata-Flavor: Google")

answered Aug 02 '21 at 15:04

jabbson

4,390
1
13
23

thanks @jabbson. I downloaded the python script. But looks like I am still missing something. – Jasmine Aug 03 '21 at 03:40
well, you would need to be more specific :) You have a startup script, where you download another script and execute it, anything in the startup script logs? Errors? – jabbson Aug 03 '21 at 15:29
I have added the error screenshot in the question – Jasmine Aug 04 '21 at 04:57
I meant the logs in `/var/log/daemon.log` – jabbson Aug 04 '21 at 19:54
well, all the errors are right there on the screenshot – jabbson Aug 06 '21 at 01:07
I executed all the commands from the start script in the terminal of the VM instance. All worked fine. – Jasmine Aug 07 '21 at 10:22
I don't think spaces around the "=" would work in either terminal or bash script, remove the spaces. For example, instead of `PYTHON_FILE_PATH = $(...` do `PYTHON_FILE_PATH=$(...`, then use absolute paths to save an execute file, see if that helps or let me know what are the errors then. – jabbson Aug 07 '21 at 17:43
I think the python script is not getting executed. its not printing hello in the terminal. I tried this method `./generate_fd.py >> testpy-output.txt && echo "completed with generate_fd.py`'. the testpy-output.txt is empty. – Jasmine Aug 09 '21 at 07:00
In the screenshot you have in your first port it couldn't execute the script, because it couldn't download it, because of the "=" sign, if you have fixed the issue with spaces around "=", then show the new output. – jabbson Aug 09 '21 at 11:59

Run a python script from the startup script

1 Answers1