Terraform - ordered generation of resources which are related based on a list variable

Question

I currently try to automate nested SumoLogic forder creation as part of my custom module. I have to use this resource. I need to create a folder path similar to:

parent_folder_path        = "SRE/Test/Troubleshooting"

and due to the fact that this variable will change between environments I cannot hardcode creation of the underlying resources. The problematic part is that all shown folders (SRE, Test, Troubleshooting) need to be created in a sequence because the latter needs id of the former (eg. Test folder needs id of already created SRE folder) to be created.

The end result at which I am aiming is automatically generated code as below:

resource "sumologic_folder" "SRE" {
  provider    = sumologic
  name        = "SRE"
  description = ""
  parent_id   = "0000000000XXXXX"
}
resource "sumologic_folder" "Test" {
  provider    = sumologic
  name        = "Test"
  description = ""
  parent_id   = sumologic_folder.SRE.id
}
resource "sumologic_folder" "Troubleshooting" {
  provider    = sumologic
  name        = "Troubleshooting"
  description = ""
  parent_id   = sumologic_folder.Test.id
}

I tried an approach which uses templatefile() and local_file:

parent_directories.tftpl

%{~ for index, path_part in parent_folder_path ~}
%{~ if index == 0 ~}
resource "sumologic_folder" "${replace(path_part, " ", "_")}" {
  provider    = sumologic
  name        = "${path_part}"
  description = ""
  parent_id   = "${root_folder_id}"
}
%{~ else }
resource "sumologic_folder" "${replace(path_part, " ", "_")}" {
  provider    = sumologic
  name        = "${path_part}"
  description = ""
  parent_id   = sumologic_folder.${replace(parent_folder_path[index - 1], " ", "_")}.id
}
%{~ endif ~}
%{~ endfor ~}

main.tf

resource "local_file" "parent_directories" {
  content              = templatefile("${path.module}/parent_directories.tftpl", { parent_folder_path = split("/", var.parent_folder_path), root_folder_id = var.root_folder_id })
  filename             = "${path.module}/parent_directories.tf"
}

and the file was correctly generated during terraform apply run but I was not able to include it in the scope of the run dynamically.

Does anyone know how to handle such usecase?

Thanks in advance for all help.

Best Regards, Rafal.

I think your problem is mostly because of generating `.tf` files on the fly using same Terraform. I.e. some kind of meta-programming. In my mind, Terraform HCL is not really convenient for usual kind of programming, not to say meta-programming. — Grzegorz Oledzki, Dec 02 '22 at 14:17
As written these appear to be two different questions. The first question is easily achieved intrinsically in Terraform, but is unclear what the desired ordering of the resources should be and why. The second question about generating Terraform configs with the template rendering engine would only be possible with `-target` and two consecutive executions, but it is unclear why you want to want to do this in the first place. — Matthew Schuchard, Dec 02 '22 at 14:21
Agreed with both points. You could use different approaches, e.g., terraform workspaces or the `for_each` meta-argument. Templating terraform with terraform sounds like an anti-pattern. — Marko E, Dec 02 '22 at 14:26
I don't think this use case can be solved in terraform resources that refer back to them saelf will always create a cycle error even if you try using short circuit logic like `parent_id = each.value["parent"] == null ? local.root_parent : sumologic_folder.folders[each.value["parent"]]` the latter option will still be evaluated by the terraform graph even when it might never be called which will result in a cycle error `Error: Cycle: sumologic_folder.folders["Troubleshooting"], sumologic_folder.folders["SRE"], sumologic_folder.folders["Test"]` — Chris Doyle, Dec 02 '22 at 15:16

score 0 · Answer 1 · answered Dec 04 '22 at 23:06

I understand what you are trying to achieve - you want to create multiple resources of the same type, but relying on each one of them created before (previous one on the list), at the same time not knowing how many there would be (more folders in path). I am afraid it is not how Terraform works. You would create a cycle between the list or map of the same resources.

That said, however, I can offer you the ugly solution. If you can limit to some number of subdirectories, let's say up to five or ten levels, you can do that in the code that will create three folders if there are three dirs in the path, and four if there are four, and so on. You just stop creating resources if this level is empty.

Let's say you have a sumo module:

variable "parent_path" {}
variable "name" {}

data "sumologic_folder" "parent" {
  path = var.parent_path
}

resource "sumologic_folder" "folder" {
  provider    = sumologic
  name        = var.name
  description = ""
  parent_id   = data.sumologic_folder.parent.id
}

output "path" {
  value = "${var.path}/${var.name}"
}

And then you can split the path to the list of folders and create as many resources as there are folders in the path, for example: AA/BB/CC/DD = 4 sumofolders.

locals {
  desired_path = "SRE/Test/Troubleshooting" # example - 3 folders
  regex        = regexall("[^//]+", local.desired_path)
  path0        = "/"
}

module "sumo" {
  source = "./sumo"
  name   = local.regex[0]
  parent_path   = local.path0 # var.parent_path
}

module "sumo_child_1" {
  source = "./sumo"
  count = try(local.regex[1], null) == null ? 0 : 1
  name   = try(local.regex[1], "none")
  parent_path   = module.sumo.path
}

module "sumo_child_2" {
  source = "./sumo"
  count = try(local.regex[2], null) == null ? 0 : 1
  name   = try(local.regex[2], "none")
  parent_path   = module.sumo_child_1.path
}

module "sumo_child_3" { # this is NOT going to be even created in our example
  source = "./sumo"
  count = try(local.regex[3], null) == null ? 0 : 1
  name   = try(local.regex[3], "none")
  parent_path   = module.sumo_child_2.path
}

# and so on... if there are no more folders in the path, the resources won't be created anyway.

Now let me say that again, this is a very ugly solution... but it works. Cheers.

Terraform - ordered generation of resources which are related based on a list variable

1 Answers1