How to launch GCP Compute Engine VM with Startup Script via Python API?

Question

GCP has a published create_instance() code snippet available here, which I've seen on SO in a couple places e.g. here. However, as you can see in the first link, it's from 2015 ("Copyright 2015 Google Inc"), and Google has since published another code sample for launching a GCE instance dated 2022. It's available on github here, and this newer create_instance function is what's featured in GCP's python API documentation here.

However, I can't figure out how to pass a startup script via metadata to run on VM startup using the modern python function. I tried adding

    instance_client.metadata.items = {'key': 'startup-script',
                                      'value': job_script}

to the create.py function (again, available here along with supporting utility functions it calls) but it threw an error that the instance_client doesn't have that attribute.

GCP's documentation page for starting a GCE VM with a startup script is here, where unlike most other similar pages, it contains code snippets only for console, gcloud and (REST)API; not SDK code snippets for e.g. Python and Ruby that might show how to modify the python create_instance function above.

Is the best practice for launching a GCE VM with a startup script from a python process really to send a post request or just wrap the gcloud command

gcloud compute instances create VM_NAME \
  --image-project=debian-cloud \
  --image-family=debian-10 \
  --metadata-from-file=startup-script=FILE_PATH

...in a subprocess.run()? To be honest I wouldn't mind doing things that way since the code is so compact (the gcloud command at least, not the POST request way), but since GCP provides a create_instance python function I had assumed using/modifying-as-necessary that would be the best practice from within python...

Thanks!

Answer 1

So, the simplest (!) way with the Python library to create the equivalent of --metadata-from-file=startup-scripts=${FILE_PATH} is probably:

from google.cloud import compute_v1

instance = compute_v1.Instance()

metadata = compute_v1.Metadata()
metadata.items = [
    {
        "key":"startup-script",
        "value":'#!/usr/bin/env bash\necho "Hello Freddie"'
    }
]

instance.metadata = metadata

And another way is:

metadata = compute_v1.Metadata()

items = compute_v1.types.Items()
items.key = "startup-script"
items.value = """
#!/usr/bin/env bash

echo "Hello Freddie"
"""

metadata.items = [items]

NOTE In the examples, I'm embedding the content of the FILE_PATH in the script for convenience but you could, of course, use Python's open to achieve a more comparable result.

It is generally always better to use a library|SDK if you have one to invoke functionality rather than use subprocess to invoke the binary. As mentioned in the comments, the primary reason is that language-specific calls give you typing (more in typed languages), controlled execution (e.g. try) and error handling. When you invoke a subprocess its string-based streams all the way down.

I agree that the Python library for Compute Engine using classes feels cumbersome but, when you're writing a script, the focus could be on the long-term benefits of more explicit definitions vs. the short-term pain of the expressiveness. If you just wanna insert a VM, by all means using gcloud compute instances create (I do this all the time in Bash) but, if you want to use a more elegant language like Python, then I encourage you to use Python entirely.

CURIOSITY gcloud is written in Python. If you use Python subprocess to invoke gcloud commands, you're using Python to invoke a shell that runs Python to make a REST call ;-)