Terraform - loop over template

Question

I'm new to terraform and I try to achieve something that could remove the redundoncy of my infrastructure. Basically I have one module that calls child modules like so :

module "jobs" {
  source = "./jobs"

  environment                     = var.environment
  region                          = var.region
  name                            = var.name
  bucket_id                       = module.commons.bucket_id
  bucket_name                     = module.commons.bucket_name
}

Inside the jobs folder, I have multiples files that does the same thing but with different values like so :

resource "aws_s3_object" "test" {
  bucket = var.bucket_id
  key    = "scripts/test_samples.sql"
  source = "../src/main/resources/test_samples.sql"

}

resource "aws_s3_object" "test_folder" {
  bucket = var.bucket_id
  key    = "raw/test_samples/"
}

resource "aws_glue_job" "test_samples" {
  name     = "test_samples"
  role_arn = var.glue_role_arn
  execution_class = "STANDARD"

  execution_property {
    max_concurrent_runs = 1
  }

  command {
    name            = "glueetl"
    script_location = "s3://${var.bucket_name}/${aws_s3_object.simple_jdbc_code.key}"
  }

  connections = [var.databricks_jdbc_connection_name]

  max_retries = 0
  number_of_workers = 2
  worker_type = "G.1X"
  timeout = 10 # minutes
  glue_version = "3.0"
  
etc etc.
}

resource "aws_glue_trigger" "test_schedule" {
  name     = "test_samples"
  schedule = "cron(00 02 ? * SUN *)"
  type     = "SCHEDULED"

  actions {
    job_name = aws_glue_job.test_samples.name
  }
}

Basically, instead of running the exact same file with different values, I would want to create a sort of template of the previous code, applying the same parameters but with different values (a sort of loop that would read some differents set of parameters during the execution).

Is there an easy way to realize that ? I've heard of Terragrunt which can help to factorize some terraform code, can I use it in this case ?

Thanks in advance !

You need to provide more info. What exactly is different between those files? What parameters do you change and how? — Marcin, Oct 25 '22 at 10:56
The template is not related to the S3 object nor something else. @Marcin : The only differences between those files would be the name/key/source sql file of the resources that I would want to variabilize. In my mind, I would want to be able to execute the "template" file with different set of variables so that whenever I want to add a new source, I would just need to add new values in a file to set the variables. — Meyer Cohen, Oct 25 '22 at 11:20
No, test_samples.sql would be replaced by a variable. Like we would have for the source of the first resource : source = "../src/main/resources/${var.something}" and what I want is being able to store somewhere a sort of list for the something variable and when I'll do a terraform apply, I would want the template to be executed as many time as the size of my list and each time it executes the template it replaces with one the value of the list. — Meyer Cohen, Oct 25 '22 at 12:06
Ok, so each of the values from the `var.something` would correspond to an actual file? Would those files have to be templated as well or you would have them ready? — Marko E, Oct 25 '22 at 12:12
Yes exactly, each values corresponds to an actual file. Those files are already ready. — Meyer Cohen, Oct 25 '22 at 12:18

score 2 · Accepted Answer · answered Oct 25 '22 at 12:27

Based on the discussion in the comments, you could create a variable on the module level:

variable "something" {
  type        = list(string)
  description = "List of SQL files to be used."
}

Then, in the module itself you would use a combination of for_each [1] and toset [2] built-in functions:

resource "aws_s3_object" "test" {
  for_each = toset(var.something)
  bucket   = var.bucket_id
  key      = "scripts/${each.key}"
  source   = "../src/main/resources/${each.key}"

}

When calling the module, you would provide the list of file names:

module "jobs" {
  source = "./jobs"

  environment                     = var.environment
  region                          = var.region
  name                            = var.name
  bucket_id                       = module.commons.bucket_id
  bucket_name                     = module.commons.bucket_name
  something                       = ["test_samples1.sql", "test_samples2.sql", "test_samples3.sql"]
}

[1] https://developer.hashicorp.com/terraform/language/meta-arguments/for_each

[2] https://developer.hashicorp.com/terraform/language/functions/toset

Terraform - loop over template

1 Answers1