Create EBS snapshot using Terraform and count()

Question

I want to create snapshots for all gp2 type EBS volumes. I have the following code:

data "aws_ebs_volume" "ebs_volumes" {
  filter {
    name   = "volume-type"
    values = ["gp2"]
  }
}

resource "aws_ebs_snapshot" "ebs_snapshot" {
  count = "${length(data.aws_ebs_volume.ebs_volumes.ids)}"
  volume_id = "${element(data.ebs_volume.ebs_volumes.ids, count.index)}"
}

Instead I got the following error:

terraform plan

Error: resource 'aws_ebs_snapshot.ebs_snapshot' config: unknown resource 'data.ebs_volume.ebs_volumes' referenced in variable data.ebs_volume.ebs_volumes.ids

Since I am using data source, the list returns by data.aws_ebs_volume is dynamic - not a static variable as discussed in Create snapshots of multiple EBS volumes using Terraform.

Using terraform show:

data.aws_ebs_volume.ebs_volumes:
  id = vol-00b3eaaf04b9377cb
  arn = arn:aws:ec2:us-east-1:ACCOUNT_ID:volume/vol-00b3eaaf04b9377cb
  availability_zone = us-east-1c
  encrypted = false
  filter.# = 1
  filter.3737401200.name = volume-type
  filter.3737401200.values.# = 1
  filter.3737401200.values.0 = gp2
  iops = 100
  kms_key_id =
  most_recent = false
  size = 8
  snapshot_id = snap-01d81204beb02804b
  tags.% = 0
  volume_id = vol-00b3eaaf04b9377cb
  volume_type = gp2

Is there a reason you want to do this directly with Terraform rather than use AWS Backup or AWS' Data Lifecycle Manager? — ydaetskcoR, Feb 17 '19 at 17:30
Hi. Yes. Lifecycle only takes snapshots every 12/24 hours. I am writing a test that could use TF and am interested in playing around with TF. Thanks for asking! — CppLearner, Feb 17 '19 at 17:31
Can you maybe expand your question to include the things you would do to target each volume (eg how would you filter for the volumes you want to snapshot?) even if you had to do it statically and then explain how you want to be able to do it dynamically? — ydaetskcoR, Feb 17 '19 at 17:47
@ydaetskcoR Hi. I edited my question, not sure if that's more explicit. Since there are hundred of machines, I need to compute the list of EBS volumes on demand. — CppLearner, Feb 17 '19 at 17:54
It's probably worth mentioning that DLM now supports running snapshots every 2 hours: https://aws.amazon.com/about-aws/whats-new/2019/02/amazon-data-lifecycle-manager-adds-support-for-shorter-backup-intervals/ — ydaetskcoR, Feb 22 '19 at 11:38

ydaetskcoR · Accepted Answer · 2019-02-22T11:38:03.803

There are normally 2 types of data source in Terraform providers, singular ones and plural ones such as aws_ami vs aws_ami_ids with the plural one normally just returning a list of IDs of resources and the singular one offering more information about each specific resource.

Unfortunately the AWS provider has not yet implemented a plural data source for EBS volumes so you are limited to just the singular aws_ebs_volume data source and can't just dynamically return all EBS volumes matching some criteria (such as them being GP2).

As a short term fix, and if you really want to use Terraform to manage creating snapshots directly, you could use an external data source to get the list of EBS volumes outside of Terraform.

A quick example of this might look something like this (untested):

data "external" "all_gp2_ebs_volumes" {
  # Should run something like:
  # `aws ec2 describe-volumes --query 'Volumes[].VolumeId' --filters Name=volume-type,Values=gp2`
  # but return the result in a way that the data source needs it.
  program = ["python", "${path.module}/get-all-gp2-ebs-volumes.py"]
}

resource "aws_ebs_snapshot" "ebs_snapshot" {
  count     = "${length(data.external.all_gp2_ebs_volumes.result.ids)}
  volume_id = "${data.external.all_gp2_ebs_volumes.result.ids[count.index]}"
}

Unless the 2 hour period restriction on AWS DLM or AWS Backup is a real blocker for you for some reason I would use those. There is already an aws_dlm_lifecycle_policy resource to configure this via Terraform and then this issue is tracking the creation of the necessary AWS Backup resources.

Another alternative would be to raise a feature request on the issue tracker for a aws_ebs_volume_ids data source (I couldn't find an existing feature request from a quick look).

Thank you very much. I did not know the difference of the two data sources, I was always puzzled by the two. Good knowledge to have. I will work around. — CppLearner, Feb 17 '19 at 18:05

Create EBS snapshot using Terraform and count()

1 Answers1