3

I've been taking a look on how to modify the default values that EMR gives to the cluster depending on the type of machine it is.

In my case, it's a pretty basic setup of a m4.large as master and c3.2xlarge as core and the same for the task. The value that EMR sets for this property in the Task is 2880 (from http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/TaskConfiguration_H2.html). However, I need to increase this default value.

I've been trying to change this default configuration in my Cloud Formation template but for some reason it doesn't accept it. This is what I'm adding to the CF template, I can't seem to find any examples of this online:

    "Configurations" : [{
      "Classification": "mapred-site",
      "Properties": {
        "mapreduce.reduce.memory.mb": "4096"
    }}]

Any help would be greatly appreciated!

Essex
  • 6,042
  • 11
  • 67
  • 139
jonordona
  • 43
  • 6
  • If an answer below solved your question, don't forget to [mark it as Accepted](http://meta.stackexchange.com/a/5235/327137). If your question is still not answered, please provide additional details so others can help better answer your question. – wjordan Jan 07 '17 at 20:08
  • @wjordan thanks, you just earned me extra some points. =D – Kristian Jan 24 '17 at 18:59

1 Answers1

1

I had a similar problem, the solution for me was the change the key name from Properties to ConfigurationProperties because the syntax in CF specifically seems to be a bit different than you'd use within EMR's config section.

Try this:

"Configurations": [
  {
      "Classification": "mapred-site",
      "ConfigurationProperties": {
        "mapreduce.reduce.memory.mb": "4096"
      },
      "Configurations": []
  }
]
Kristian
  • 21,204
  • 19
  • 101
  • 176
  • 1
    Thanks! This actually worked, but I found out that for my specific case, I needed the default value to stay low, but only be set to 4096 on some jobs. I discovered how to set this in the application specific scenario I'm using (Druid batch ingestion). – jonordona Jan 24 '17 at 14:49