4

When a cloud function is deployed I assume Google performs some equivalent of:

pip install -r requirements.txt

Let's say the requirements.txt file contains:

google-cloud-pubsub
google-cloud-storage==1.26.0

Since the cloud storage version is specified, all existing and future scaled function instances will install that version.

A gcloud functions deploy ... would install 1.26.0 storage libraries on all instances.

However, let's say there is an instance already running with google-cloud-pubsub version 1.0.2 installed and the newest version is 1.3.0.

The pip command above would not load a newer version.

Instead there would be a Requirement already satisfied response. The existing instance would stay on version 1.0.2 while any newly scaled instances would pull 1.3.0. There would be a mismatch of library versions across instances of the same cloud function.

  1. Am I understanding this process accurately? Does GCP do an equivalent of pip install -r requirements.txt?
  2. Is there a way to force cloud functions to import the newest version of a library during deployment via requirements.txt or otherwise?
Dustin Ingram
  • 20,502
  • 7
  • 59
  • 82
ProGirlXOXO
  • 2,170
  • 6
  • 25
  • 47

2 Answers2

3
  1. Am I understanding this process accurately? Does GCP do an equivalent of pip install -r requirements.txt?

You are correct. On deploy, Cloud Functions builds an image, installs your dependencies and your function into it, and uses that to serve your function.

  1. Is there a way to force cloud functions to import the newest version of a library during deployment via requirements.txt or otherwise?

I'm not sure I'm following exactly, but you seem to think repeated installs would reuse the same image. They do not: each time your source changes and you deploy, it starts from scratch and performs the steps above.

This means that if you unpin your dependencies (i.e. no ==1.26.0) you will always get the latest compatible version of these projects every time you run a deployment.

NOTE: If your source files haven't changed, or if you are just modifying or adding environment variables, Cloud Functions will cache the previous build and will not re-evaluate your dependencies. Currently you must modify your source files to force Cloud Functions to rebuild your function.

Dustin Ingram
  • 20,502
  • 7
  • 59
  • 82
  • Thanks! Can you point to documentation that says all instance images are rebuilt on deploy? A Google Cloud support engineer suggested I pin the version to ensure we're on the latest so that threw me off. – ProGirlXOXO Feb 26 '20 at 02:07
  • https://cloud.google.com/functions/docs/concepts/exec#multiple_functions says "Every deployed function is isolated from all other functions—even those deployed from the same source file. In particular, they don’t share memory, global variables, file systems, or other state." – Dustin Ingram Feb 26 '20 at 02:45
  • 1
    Usually version pinning is motivated by _not_ unintentionally being upgraded to the latest version, which can cause breakages. It's considered best practice to use version pins. – Dustin Ingram Feb 26 '20 at 02:46
  • That link does not explain the deployment steps. It says functions are isolated. We're talking about multiple (scaled) instances of the same deployed function. It doesn't explain how a deployment affects new or existing instances of the same function. – ProGirlXOXO Feb 26 '20 at 19:56
1
  1. Is there a way to force cloud functions to import the newest version of a library during deployment via requirements.txt or otherwise?

I tested this and found the opposite of https://stackoverflow.com/a/60405672/1803897

You have to specify the newer version in requirements.txt to force the use of the newer library. Else the existing library will be used.

https://cloud.google.com/functions/docs/concepts/exec#multiple_functions does not say that some files are not reused when making a new instance. They will copy in some some files, as they will surely not make a new complete install for every instance. That would take too much time. Regarding python libraries the same point is valid. It would take too much time to install all python libraries at every deploy.

Roland Puntaier
  • 3,250
  • 30
  • 35
  • How did you test? – ProGirlXOXO Apr 14 '20 at 02:41
  • I deployed with ``mypkg>=x.y.z`` in `requirements.txt` and checked whether my changes in `mypkg` were online. They were not online. Then I changed to ``mypkg==x.y.Z`` and deployed again. Now my changes in `mypkg` were online. Afterwards I always changed `==x.y.z` to the newest version of `mypkg`. Currently I have a situation, where even so the changes did not reflect after a deploy. That can have other reasons: Maybe the upgrade of `mypkg` failed. I'm investigating. – Roland Puntaier Apr 14 '20 at 10:33
  • ... the newest `mypkg` was there all right, but the new code missed a new dependency. – Roland Puntaier Apr 14 '20 at 12:45
  • My solution was to version pin everything exactly. I tried to test. All my function did was print the version of an installed library. I set requirements to pin an older version and deployed. Then I set requirements to have that same library with no version specified. The next time the function ran it had the most up-to-date library version. I was only running the function once per minute though so it probably did not have more than one clone. I also think it would be a waste for Google to rebuild clones when requirements are already "matching". – ProGirlXOXO Apr 17 '20 at 01:54
  • @RolandPuntaier I think you're likely hitting an edge case where Cloud Functions will cache the previous build entirely if none of the source files (including dependency files) have changed. I've updated [my answer](https://stackoverflow.com/a/60405672/328036) to reflect this. – Dustin Ingram Dec 17 '20 at 05:26