How to assign a JSON file as STEP in an EMR cluster in Terraform?

Question

I'm building an EMR cluster in Terraform and in the STEP argument i want to load a JSON file that describes the list of steps.

I tried this in my main.tf :

ressource "aws_emr" "emr" {
  ...
  ...
  step = "${data.template_file.steps.rendered}"
}

data "template_file" "steps" {
  template = "${file("${path.module}/steps.json")}"
}

And this is the JSON file :

[
{
    "action_on_failure" : "CONTINUE",
    "name"              : "step_name",
    "hadoop_jar_step" : {
        "jar" : "command-runner.jar",
        "args" : [
            "spark-submit",
            "s3://mybucket/src/pyspark/script1.py",
            "1",
            "68465131321321",
            "s3://mybucket/parquet",
            "s3://mybucket/result",
            "321",
            "65165165468587",
            "654"
        ]
    }
}
]

But when i do terraform plan i got this error :

Inappropriate value for attribute "step": list of object required.

What's the problem ?

Thanks for your help.

score 1 · Answer 1 · answered Jun 13 '19 at 12:28

1

Ok, i found the solution in another website and i'll post it here maybe it will help somebody oneday

   resource "aws_emr_cluster" "cluster" {
  ...
  dynamic "step" {
    for_each = jsondecode(templatefile("steps.json", {}))
    content {
      action_on_failure = step.value.action_on_failure
      name              = step.value.name
      hadoop_jar_step {
        jar  = step.value.hadoop_jar_step.jar
        args = step.value.hadoop_jar_step.args
      }
    }
  }
  ...
}

So that way it's possible to use a JSON file a source for the step in an EMR ressource.

answered Jun 13 '19 at 12:28

user1297406

1,241
1
18
36

What is the structure for the steps.json file? – jmng Nov 27 '19 at 11:58
1

Here is an example of a steps json file : https://jsonblob.com/1007d06f-111e-11ea-952e-a98fd6b9f322 – user1297406 Nov 27 '19 at 14:00
Thanks; now I got it, it's further explained at https://docs.aws.amazon.com/cli/latest/reference/emr/add-steps.html – jmng Nov 27 '19 at 14:08

score 0 · Answer 2 · answered Jun 12 '19 at 07:44

0

Steps are not defined as JSON in the aws_emr resource, instead try:

step {
  action_on_failure = "CONTINUE"
  name              = "step_name"

  hadoop_jar_step {
    jar  = "command-runner.jar"
    args = ["spark-submit", ...]
  }
}

answered Jun 12 '19 at 07:44

Blokje5

4,763
1
20
37

That way it woks, but i'm looking for a way to configure steps through a config file like a JSON file. Is it possible ? – user1297406 Jun 12 '19 at 07:48
No, the step is only defined as a `TypeList` in the AWS provider, there is no option for JSON in there. What you could do though is to use the new `jsondecode` function in terraform 0.12: https://www.terraform.io/docs/configuration/functions/jsondecode.html. There are options in earlier versions as well, they do become a bit more hacky. – Blokje5 Jun 12 '19 at 07:56
something like that : step = jsondecode("${data.template_file.steps.rendered}") ? – user1297406 Jun 12 '19 at 08:00
Tried it but i got this error : Inappropriate value for attribute "step": element 0: attribute "action_on_failure": string required. – user1297406 Jun 12 '19 at 08:01

How to assign a JSON file as STEP in an EMR cluster in Terraform?

2 Answers2