Adding new AWS EBS Volume to ASG in same AZ

Question

ok, so I am trying to attach an EBS volume which I have created using Terraform to an ASG's instance using userdata, but now issue is both are in different AZ's, due to which, it failing to attach. Below is the steps I am trying and failing:

resource "aws_ebs_volume" "this" {
  for_each = var.ebs_block_device
    size              = lookup(each.value,"volume_size", null)
    type              = lookup(each.value,"volume_type", null)
    iops              = lookup(each.value, "iops", null)
    encrypted         = lookup(each.value, "volume_encrypt", null)
    kms_key_id        = lookup(each.value, "kms_key_id", null)
    availability_zone = join(",",random_shuffle.az.result)
}

In above resource, I am using random provider to get one AZ from list of AZs, and same list is provided to ASG resource below:

resource "aws_autoscaling_group" "this" {
  desired_capacity          = var.desired_capacity
  launch_configuration      = aws_launch_configuration.this.id
  max_size                  = var.max_size
  min_size                  = var.min_size
  name                      = var.name
  vpc_zone_identifier       = var.subnet_ids // <------ HERE
  health_check_grace_period = var.health_check_grace_period
  load_balancers            = var.load_balancer_names
  target_group_arns         = var.target_group_arns

  tag {
    key                 = "Name"
    value               = var.name
    propagate_at_launch = true
  }
}

And here is userdata which I am using:

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`

instanceId = curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id

aws ec2 attach-volume --volume-id ${ebs_volume_id} --instance-id $instanceId --device /dev/nvme1n1

Above will attach the newly created volume, as I am passing output ${ebs_volume_id} of above resource.

But, its failing because instance and volume are in different AZs.

Can anyone help me on this as a better solution than hardcoding AZ on both ASG and Volume?

Why are you creating the EBS volume this way instead of configuring the EBS volume in the launch configuration? — Mark B, Sep 16 '20 at 15:35
So when I change anything in userdata/lauch configuration, it doesn't delete/recreate volume — Ash Singh, Sep 16 '20 at 17:03

Geoff · Answer 1 · 2020-10-09T22:35:06.563

I'd have to understand more about what you're trying to do to solve this with just the aws provider and terraform. And honestly, most ideas are going to be a bit complex.

You could have an ASG per AZ. Otherwise, the ASG is going to select some AZ at each launch. And you'll have more instances in an AZ than you have volumes and volumes in other AZs with no instances to attach to.

So you could create a number of volumes per az and an ASG per AZ. Then the userdata should list all the volumes in the AZ that are not attached to an instance. Then pick the id of the first volume that is unattached. Then attach it. If all are attached, you should trigger your alerting because you have more instances than you have volumes.

Any attempt to do this with a single ASG is really an attempt at writing your own ASG but doing it in a way that fights with your actual ASG.

But there is a company who offers managing this as a service. They also help you manage them as spot instances to save cost: https://spot.io/

The elastigroup resource is an ASG managed by them. So you won't have an aws asg anymore. But they have some interesting stateful configurations.

We support instance persistence via the following configurations. all values are boolean. For more information on instance persistence please see: Stateful configuration persist_root_device - (Optional) Boolean, should the instance maintain its root device volumes. persist_block_devices - (Optional) Boolean, should the instance maintain its Data volumes. persist_private_ip - (Optional) Boolean, should the instance maintain its private IP. block_devices_mode - (Optional) String, determine the way we attach the data volumes to the data devices, possible values: "reattach" and "onLaunch" (default is onLaunch). private_ips - (Optional) List of Private IPs to associate to the group instances.(e.g. "172.1.1.0"). Please note: This setting will only apply if persistence.persist_private_ip is set to true

stateful_deallocation {
     should_delete_images              = false
     should_delete_network_interfaces  = false
     should_delete_volumes             = false
     should_delete_snapshots           = false
}

This allows you to have an autoscaler that preserves volumes and handles the complexities for you.

Adding new AWS EBS Volume to ASG in same AZ

1 Answers1