How do you manage image versions with Packer and Terraform?

Question

I'm currently using a Kubernetes cluster running on bare metal nodes provisioned with Ansible. There are plans to move to the cloud and I'm reading about Terraform and Packer, in preparation for this. Leaving the data migration aside, it seems like there is a pretty straight forward migration path for us:

1. Build image with Packer using our existing Ansible scripts
2. Deploy built image to the cloud with Terraform
3. Deploy our Kubernetes resources with our current tooling

That's all great. We now have immutable infrastructure, using state of the art tooling.

What I am struggling to find is how images built with Packer are versioned. Somewhere down the line, we will have to upgrade some software in those images. Sometimes the Ansible scripts will change, but sometimes it's just a matter of having the latest security updates in the image. Either way, Packer will have to build a new image for us and we will have to deploy it with Terraform. If the new image ends up causing trouble, we will have to revert to the old one.

I can imagine how this could be done manually by editing the template before running it and then editing the terraform configuration to pick up the new version, but that's not going to work for a CI/CD pipeline. Another issue is that we may move between different regions and vendors. So a version of the image may be present in one region, but not the other, and ideally, the pipeline should create the image if it doesn't exist and use the existing one if it is already there. This may cause images in different regions or clouds to be different, especially since they might be built on different days and have different security updates applied.

All of this is built in to the Docker workflow, but with Packer, it's far from obvious what to do. I haven't found any documentation or tutorials that cover this topic. Is there any built-in versioning functionality in Packer and Terraform? Is Terraform able to invoke Packer if an image is missing? Is there any accepted best practice?

I can imagine automating this by using the API from the cloud provider to check for the existence of the required images and invoke Packer for any missing images, before executing Terraform. This would work, but I wouldn't want to write custom integration for each cloud provider and it sounds like something that should already be provided by Terraform. I haven't used Terraform before, so maybe I just don't know where to look, and maybe it's not that difficult to implement in Terraform, but then why aren't there any tutorials showing me how?

Answer 1

This is largely provider dependent and you haven't specified the cloud provider you are using but AWS makes a good example use case here.

Terraform and Packer both have a way of selecting the most recent AMI that matches a filter.

Packer's AWS AMI builder uses source_ami_filter that can be used to select the most recent image to base your image from. An example is given in the amazon-ebs builder documentation:

{
  "source_ami_filter": {
    "filters": {
      "virtualization-type": "hvm",
      "name": "ubuntu/images/\*ubuntu-xenial-16.04-amd64-server-\*",
      "root-device-type": "ebs"
    },
    "owners": ["099720109477"],
    "most_recent": true
  }
}

A typical case here is to always use the latest official Ubuntu image to build from. If you are producing multiple AMIs for different use cases (eg Kubernetes worker nodes vs etcd nodes) then you can then build up from there so you create a golden base image with a known naming scheme (eg ubuntu/20.04/base/{{isotime | clean_resource_name}}) that has everything you want in every AMI you produce and then other AMIs can also use the source_ami_filter to select for the most recent base AMI that you have published.

Terraform's AWS provider has the aws_ami data source that works in the same way and can be used to automatically select the latest AMI that matches a filter so that publishing a new AMI and then running Terraform will generate a plan to replace your instance or launch configuration/template that is referencing the AMI data source.

An example is given in the aws_instance resource documentation:

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

resource "aws_instance" "web" {
  ami           = "${data.aws_ami.ubuntu.id}"
  instance_type = "t2.micro"

  tags = {
    Name = "HelloWorld"
  }
}

In general you should be relying on mechanisms like these to automatically select the most recently published AMI and use that instead of hardcoding AMI IDs in code.

---

Managing the lifecycle of images is beyond the scope of Packer itself and it should be used as part of a larger system. If you want to rollback an image then you have two options that are available to you:

• If your build is reproducible and the issue is in that reproducible thing then you can build and register a new image with the old code so that your newest image is now the same as the one 2 images ago
• Deregister the newest image so you will begin picking up the old image again when searching for the latest. This varies depending on cloud provider but with AWS can be done programatically such as via the aws ec2 deregister-image command line

While Packer can automatically copy images to different regions (see ami_regions for AWS) and different accounts (use ami_users to share the created AMI with the other account or a post processor to create separate copies in different accounts) it can't easily do things conditionally without you having different Packer config files for each combination of ways you want to share things and can't separate out the roll out so you release to a non production account before you release to a production account etc.

If you want to roll out AMIs in some accounts and regions but not all then you are going to need to put that logic in a higher order place such as an orchestration mechanism like your CI/CD system.

Answer 2

I wrote a blog post on this topic, Keeping Packer and Terraform images versioned, synchronized and DRY.

In summary:

Our goals

• Images built by Packer must be versioned.
• Image versioning must be DRY (kept in a single place) and shared between Packer and Terraform, to avoid going out-of-sync by mistake.
• The configuration files for Packer, Terraform and image versioning information must be stored in git, so that checking out a specific commit and doing a terraform apply should be enough to perform a rollback.
• Terraform must detect automatically, based only on local information, that there is a more recent version of one or multiple images, OR that a more recent version should be built.
• It must be possible to have N independent development/staging environments, where image versioning is independent from production.
• Approach must be IaaS agnostic (must work with any Cloud provider).

Summary of the approach

Use a naming convention like

<IMAGE-NAME> ::= <ROLE>__<BRANCH>-<REVISION>

Define the value of the variable in a separate file, packer/versions.pkvars.hcl:

service-a-img-name = "service-a__main-3"

Build the image with:

$ packer build -var-file=versions.pkrvars.hcl minimal.pkr.hcl

On the Terraform side, since file packer/versions.pkvars.hcl is in HCL, we can read it from Terraform:

$ terraform apply -var-file=../../packer/versions.pkrvars.hcl

All the details are in the blog post mentioned above.

Answer 3

So for what it’s worth, the image versioning is useful because you can save some defaults for things like kubernetes host nodes (pre-downloaded docker images, etc) so by the time it passes the AWS checks, it is already joining the cluster.

I have don’t this for numerous apps and found it is typically best to do something like below

vendor-app-appversion-epoch

This approach allows you to version your Ami along with your apps, and then you can treat your instances like cattle (to be slaughtered) versus pets (to be cared for throughout their life).

data "aws_ami" "amazon_linux2" {
  most_recent = true
  filter {
    name = "name"
    values = ["amzn2-ami-*-x86_64-gp2"]
  }

  filter {
    name = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["amazon"]
}

This will pull the latest image for linux2 when you apply terraform.