-2

Will manually Resizing a Running Cluster from AWS console use comissioning and decomissioning process internally?

We are working on EMR cluster where we resize cluster manually from aws console which leads to missing /user/oozie/share/lib/ jars some times. I explored this issue and found that Commissioning and Decommissioning of Datanode in Hadoop is the best way of resizing nodes and avoiding cluster corruption. https://acadgild.com/blog/commissioning-and-decommissioning-of-datanode-in-hadoop/ How this can be done when I am resizing cluster from aws console? Does aws console maintain this internally?

Pooja Soni
  • 137
  • 1
  • 2
  • 9

1 Answers1

0

Yes an EMR scale-down, whether it be manual of Autoscaling will invoke HDFS / YARN decommissioning process before terminating a node. This will ensure the HDFS Data is not lost and running YARN jobs are not abruptly killed. see: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-scaledown-behavior.html

For scale-ups, it just a simple startup of DataNode and NM daemons with correct configurations for master to register them.

jc mannem
  • 2,293
  • 19
  • 23