0

We have a managed instance group on GCP which is configured with an autoscaling rule.

The instance group references an instance template that was created via gcloud compute instance-templates create-with-container. The container image is hosted on GCR.

I'm trying to understand the best way to deploy frequent updates to this instance group, for example in a CI/CD pipeline.

Based on my current understanding, it seems the procedure is:

  1. Build and push a new docker image to GCR
  2. Create a new instance template.
  3. Submit a rolling update to the instance group which points at the new instance template.

However in a CI/CD pipeline, it seems:

  1. This is going to create hundreds, potentially thousands of dangling instance templates that are only used once and never again. Is there a problem with this?
  2. Its unclear how the instance templates should be named or versioned. I was thinking of writing a hash of the docker image into the instance template name when the template is created, but this seems needlessly manual.

Is this really the optimal way to deploy updates to an instance group, or am I missing something? Does deployment manager simplify this all, for example in the name generation or template cleanup?

Josh Bothun
  • 1,324
  • 9
  • 9
  • 1
    It's a bit hard to answer this as your title asks for "best practices" and the multiple questions have mostly opinion-based answers. Are you able to focus it a bit more on a more clearly defined issue? Have you tried out the scenario at hand to know if the proposed issues are in fact problems? – Andy Shinn May 07 '20 at 02:10
  • 1
    I am not aware of a best practices document for your use case. As a side note: If your containers are specified using the "latest" tag, then just replacing the instances will result in the VM containers being updated without creating new instance templates. It just depends on what level of control and rollback you require. https://cloud.google.com/sdk/gcloud/reference/compute/instance-groups/managed/rolling-action/replace – John Hanley May 07 '20 at 02:27
  • 1
    Speaking bluntly, from the question it follows that you want to streamline CI/CD pipeline: make frequent updates of dockerized apps in an environment with autoscaling. Due to the existing solution, instead of juggling light containers you have to manipulate heavy instance templates, which leads to extra costs. Based on existing limited information, this solution does not meet the project needs well. Why not consider GKE? Could you please provide more details about requirements of your project and why should you stick to the current solution so that the community can find optimal approach? – mebius99 May 08 '20 at 18:21
  • @mebius99 We chose instance groups over GKE as GKE felt like overkill for the use case - this is a very simple single service which typically runs 1 instance only in 1 region, so it didn't seem worth it. That said, if even running a single instance on GKE results in a lower overhead update path in a CI/CD pipeline, then i'll definitely look into it. – Josh Bothun May 09 '20 at 23:10

2 Answers2

1

To make behavior of deployments more predictable, it'd be better to follow "deterministic" approach that involves specifying of tags with an exact build version or hash. A consequence of this is a lot of template instances that will have to be managed later.

On the other hand this provides version control and rollback opportunity. There are the versions and instanceTemplate properties of a MIG object and container image name and tag of an instance template, that can be used for version control.

Kubernetes Engine looks axiomatic approach in cases like yours. But if GKE is not suitable due to objective factors, you can consider Cloud Run. It supports autoscaling, service revisions, and it takes care of imageDigest. As with the instance templates, configurations of each revision are immutable.

Deployment Manager strength is in building complicated deployments. With DM each build should be in a form of a separate deployment. Python-based or Jinja2-based templates can help in generating meaningful names, labels, metadata, that can be usable during clean up later on. But deployments themselves have to be managed and cleaned up anyway: with gcloud deployment-manager deployments delete.

Whatever you prefer, there's a need for sorting and cleaning obsolete "managed items": instance templates, revisions or deployments. This is up to the user, Google does not provide services for lifecycle management.

A primitive lifecycle solution could be gcloud called from a cron job running on a VM with a service account that is granted appropriate permissions. Managed items could be requested based on key attributes such as ID, timestamp, hash, digest. Lists of items can be sorted by timestamp or version tag so that delete, for instance, the oldest 100 items or keep the latest 100 items.

Compute Engine > Doc > Deterministic instance templates

Compute Engine > Doc > Deploying containers on VMs and MIGs > Updating a managed instance group to a new version of a container image

Compute Engine > Doc > Creating MIGs > Changing the instance template for a MIG

Compute Engine > Doc > Rolling out updates to MIGs > Relationship between versions and instanceTemplate properties for a managed instance group

mebius99
  • 2,495
  • 1
  • 5
  • 9
0

I faced the same problem and solved it by overriding instance template properties with an all-instances configuration (see link below).

This way:

  • My managed instances group is initialized the first time with an instance template with a placeholder container image.
  • My deployment pipeline is just responsible for updating the MIG with nuew versions of the container image.

As Placeholder container-image you can take for example: us-docker.pkg.dev/cloudrun/container/hello

https://cloud.google.com/compute/docs/instance-groups/set-mig-aic

I hope this helps!

Najib Bakahoui
  • 41
  • 4