Terraform stalls while trying to get IP addresses of multiple instances?

Question

So I am using terraform to provision ec2 instances and openstack instances. I am trying to reference the IP addresses of the instances I am creating because I need to run commands that use them (to set up consul). However after adding references to these variables terraform just stalls out and does absolutely nothing after I run a terraform apply or terraform plan:

Here is a sample of the resource block for what I am trying to run:

resource "aws_instance" "consul" {
  count = 3
  ami = "ami-ce5a9fa3"
  instance_type = "t2.micro"
  key_name = "ansible_aws"
  tags {
    Name = "consul"
  }

  connection {
    user = "ubuntu"
    private_key="${file("/home/ubuntu/.ssh/id_rsa")}"
    agent = true
    timeout = "3m"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y curl",
      "echo ${aws_instance.consul.0.private_ip} >> /home/ubuntu/test.txt",
      "echo ${aws_instance.consul.1.private_ip} >> /home/ubuntu/test.txt",
      "echo ${aws_instance.consul.2.private_ip} >> /home/ubuntu/test.txt"
    ]
  }
}

Update: so I tried running a similar command with my openstack cloud and got the same problem:

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y curl",
      "echo ${openstack_compute_instance_v2.consul.0.network.0.fixed_ip_v4}",
      "echo ${openstack_compute_instance_v2.consul.1.network.1.fixed_ip_v4}",
      "echo ${openstack_compute_instance_v2.consul.2.network.2.fixed_ip_v4}"
    ]
  }

So I have found if instead I use only one of the IP addresses then the other instances wont even be created until my first instance is created, like in the block below:

  provisioner "remote-exec" {
    inline = [
      "echo ${openstack_compute_instance_v2.consul.0.network.0.fixed_ip_v4}",
    ]
  }

I need all my instances to be created at the same time and have access to the IP addresses of all the other instances created as soon as they are created.

At a _very_ quick glance that looks like that should do exactly what you want. What's wrong with it? — ydaetskcoR, Jun 15 '16 at 21:55
That shouldn't be a problem. The `aws_instance.consul...` part is saying to get the output from an `aws_instance` resource called `consul`. What happens if you remove the `count` and have a single `"echo ${aws_instance.consul.private_ip} >> /home/ubuntu/test.txt`"? — ydaetskcoR, Jun 15 '16 at 22:10
So I tried doing something very similar to this with my openstack cloud and got the same exact problem. after running `terraform apply` my terminal starts doing absolutely nothing. After waiting for it to do something for like 5 min I did a double `control + c` to exit and was then presented with a wall of errors. — Alex Cohen, Jun 16 '16 at 15:22
So when I have a single instance and run `"echo ${aws_instance.consul.private_ip}"` it works fine, but with multiple instances terraform will just do absolutely nothing after I run a `terraform apply` or `terraform plan` — Alex Cohen, Jun 16 '16 at 15:57

score 4 · Accepted Answer · answered Jun 16 '16 at 17:43

So taking a longer look at this it looks like it is actually an issue with Terraform where it gets locked into a dependency cycle.

That same issue also includes a workaround for this by using a null_resource to then connect to all of the provisioned instances and run a script against them.

For your use case you could use something like this:

resource "aws_instance" "consul" {
  count = 3
  ami = "ami-ce5a9fa3"
  instance_type = "t2.micro"
  key_name = "ansible_aws"
  tags {
    Name = "consul"
  }
}

resource "null_resource" "configure-consul-ips" {
  count = 3

  connection {
    user = "ubuntu"
    private_key="${file("/home/ubuntu/.ssh/id_rsa")}"
    agent = true
    timeout = "3m"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y curl",
      "sudo echo '${join("\n", aws_instance.consul.*.private_ip)}' > /home/ubuntu/test.txt"
    ]
  }
}

This should bring up 3 instances and then connect to them, install curl and then create a file with a new line separated list of the private IP addresses of the 3 instances.

I did notice that `null resources` cant run all the commands my other servers do, so essentially it is not reconnecting to my servers created in the previous `resource` block. Another solution was to create another server after the first 3 and run remote bash commands from that new server. It was pretty easy to work that into my terraform script, since I needed to create a load balancer instance anyway. So that 4th instance runs the commands I need and has access to the IP addresses of the first 3 servers. — Alex Cohen, Jun 17 '16 at 17:49

Terraform stalls while trying to get IP addresses of multiple instances?

1 Answers1

Linked