Create Terraform resources out of JSON values

Question

I am looking for a way to generate Terraform code based on JSON values. Imagine I have a JSON file with the following structure:

{
  "settings": [
    {
      "conf": [
        {
          "setting": "DeploymentPolicy",
          "namespace": "aws:elasticbeanstalk:command",
          "value": "AllAtOnce"
        },
        {
          "setting": "BatchSize",
          "namespace": "aws:elasticbeanstalk:command",
          "value": "30"
        },
        {
          "setting": "BatchSizeType",
          "namespace": "aws:elasticbeanstalk:command",
          "value": "Percentage"
        }
      ]
    }
  ]
}

What I want to do is the following: Creating a working Terraform resource based on the JSON file values, e.g. a beanstalk environment like this:

resource "aws_elastic_beanstalk_environment" "app_prod" {
  name                   = "${aws_elastic_beanstalk_application_version.app.name}-prod"
  application            = aws_elastic_beanstalk_application.app.name
  solution_stack_name    = data.aws_elastic_beanstalk_solution_stack.latest_linux_java.name
  wait_for_ready_timeout = "10m"
  version_label          = aws_elastic_beanstalk_application_version.app.name

  # Elastic beanstalk configuration
  setting {
    name      = "DeploymentPolicy"
    namespace = "aws:elasticbeanstalk:command"
    value     = "AllAtOnce"
  }
  setting {
    name      = "BatchSize"
    namespace = "aws:elasticbeanstalk:command"
    value     = "30"
  }

  ...

}

Therefore I have to create the settings block in HCL (Terraform configuration) based on the JSON values. This means the JSON file above should result in:

  setting {
    name      = "DeploymentPolicy"
    namespace = "aws:elasticbeanstalk:command"
    value     = "AllAtOnce"
  }
  setting {
    name      = "BatchSize"
    namespace = "aws:elasticbeanstalk:command"
    value     = "30"
  }
  setting {
    name      = "BatchSizeType"
    namespace = "aws:elasticbeanstalk:command"
    value     = "Percentage"
  }

As you can see, the structure of JSON and HCL is very similar, but not identical. See e.g. settings, conf, or setting instead of name in the JSON.

A possible approach would be to read the JSON values and store them in an array or a map. But I have no idea how I could generate valid HCL and inject it in the desired part of the resource. Furthermore I tried to use a template but Terraform does not support the looping functionality that I need to iterate over the settings.

To sum up:

Input is a JSON file that must be read
JSON contains settings (besides other information)
The number of settings can differ
Somehow I have to generate a settings block
Somehow I have to inject this settings blok in the resource

Does anyone have an idea how to do that? Any other approaches? Thanks a lot!

Looks like you want to add some programming while using infrastructure as code, Check out `aws-cdk` you might find it really helpful with what you are looking for - https://docs.aws.amazon.com/cdk/latest/guide/home.html — Amit Baranes, Jan 11 '20 at 00:24
Thanks for the information. I am using terraform up to now and therefor its not exactly what I am looking for... But good to know! — Stefan, Jan 11 '20 at 19:23

score 8 · Accepted Answer · answered Jan 11 '20 at 01:12

8

Assuming that your JSON object were in a file called settings.json inside your module directory, you could do something like this:

locals {
  environment_settings = jsondecode(file("${path.module}/settings.json")).settings[0].conf[0]
}

resource "aws_elastic_beanstalk_environment" "app_prod" {
  name                   = "${aws_elastic_beanstalk_application_version.app.name}-prod"
  application            = aws_elastic_beanstalk_application.app.name
  solution_stack_name    = data.aws_elastic_beanstalk_solution_stack.latest_linux_java.name
  wait_for_ready_timeout = "10m"
  version_label          = aws_elastic_beanstalk_application_version.app.name

  dynamic "setting" {
    for_each = local.environment_settings
    content {
      namespace = setting.value.namespace
      name      = setting.value.setting
      value     = setting.value.value
    }
  }
}

This special dynamic block is a sort of macro to create repeated setting blocks, each one correlating with one element of the collection given in for_each.

You can do whatever transformations of the input you need using Terraform's expression language in the locals block to ensure that the local.environment_settings value contains one element for each setting block you will generate, and then in the content nested block tell Terraform how to populate the setting arguments based on those element values.

answered Jan 11 '20 at 01:12

Martin Atkins

62,420
8
120
138

Wow. That is exactly what I was looking for. Thank you so much! Would it even be possible to inject entire resources? Suppose the JSON file had one more level, in which several beanstalk environments are contained. I have the case that I have to import a variable number of users with different authorizations. Defining a dynamic block outside of a resource does not work. Nesting them doesn't seem to be that easy either. – Stefan Jan 11 '20 at 19:32
1

If you have a JSON file with an array of objects or an object of objects then you could massage that to be compatible with what Terraform's [resource `for_each`](https://www.terraform.io/docs/configuration/resources.html#for_each-multiple-resource-instances-defined-by-a-map-or-set-of-strings) expects and define multiple similar resource instances in the same file, indeed. There's [an example with `csvdecode`](https://www.terraform.io/docs/configuration/functions/csvdecode.html#use-with-the-for_each-meta-argument) that you could adapt to JSON instead. – Martin Atkins Jan 13 '20 at 21:06
Wow, that is amazing. Thanks for your input. I was able to use the count instead: In the Resource `count = length(local.environments)` and in the dynamics block `for_each = local.environments[count.index].settings[*]`. You shouldn't use this feature in an infaltionary manner, but it can be a very powerful tool. But more complex expressions like JSONPath are not possible, right? I know it doesn't work that way `specific_setting = jsondecode(file("${path.module}/settings.json")).environments[*].settings[?(@.name=="DeploymentPolicy")]` but it would be good to have something like that. – Stefan Jan 14 '20 at 15:02
1

It's hard to continue this conversation in detail here so might be worth a separate question if you want to dig in some more, but the Terraform feature for projecting and filtering collections is [`for` expressions](https://www.terraform.io/docs/configuration/expressions.html#for-expressions), which have some similar capabililties to JSONPath but not using JSONPath syntax directly. – Martin Atkins Jan 14 '20 at 17:40
I will have a look at it and if there are any questions I will create a separate question. Thanks so far! :) – Stefan Jan 14 '20 at 19:27

Create Terraform resources out of JSON values

1 Answers1