1

I need to make a change to the YARN configuration on an EMR cluster. Do I need to make the change to just the yarn-site.xml file on the Hadoop master ? If so, how can I propagate the change to the datanodes ? Do I just need to restart yarn as detailed here ? I am using EMR 5.8.0.

https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/

femibyte
  • 3,317
  • 7
  • 34
  • 59
  • Possible duplicate of [How to set spark.driver.memory for Spark/Zeppelin on EMR](https://stackoverflow.com/questions/47531410/how-to-set-spark-driver-memory-for-spark-zeppelin-on-emr) – eliasah Jan 18 '18 at 15:59
  • It's not an exact duplicate but it's the same idea – eliasah Jan 18 '18 at 16:00
  • Based on this thread it seems as if its not possible to push out the modified configuration to the worker nodes currently in EMR. https://stackoverflow.com/questions/43404236/how-to-change-yarn-scheduler-configuration-on-aws-emr?rq=1 – femibyte Jan 18 '18 at 16:00
  • According to my personal experience, it’s push as a configuration on cluster creation like I described in my answer – eliasah Jan 18 '18 at 16:23

1 Answers1

0

You will need to identigy which YARN Daemon enforces that parameter and if needed will need to restart that Daemon accordingly.

Ex: EMR Master has YARN ResourceManager EMR Core has YARN Nodemanager

If you need to change a parameter that corresponds to YARN ResourceManager(like yarn.resourcemanager.*), then you might need to edit yarn-site on just master and restart just the ResourceManager daemon.

If you want to change a parameter like yarn.nodemanager.* , then you will need to change yarn-site on all core nodes and might need to restart NodeManager daemon on all core nodes.

Now, when it comes to how to change this setting on all core's at once, there are lot of tools out there to do it(Like Ansible, PDSH , AWS SSM etc. ). EMR does not have any API that supports changing config's on fly. If you are trying to provision a cluster with desired configuration , use EMR Configurations API. https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html

jc mannem
  • 2,293
  • 19
  • 23