How do I troubleshoot and recover a Lost Node in my long running EMR cluster?
The node stopped reporting a few days ago. The host seems to be fine and HDFS too. I noticed the issue only from the Hadoop Applications UI.
Asked
Active
Viewed 2,515 times
1 Answers
1
EMR nodes are ephemeral and you cannot recover them once they are marked as LOST. You can avoid this in first place by enabling 'Termination Protection' feature during a cluster launch.
Regarding finding reason for LOST node, you can probably check YARN ResourceManager logs and/or Instance controller logs of your cluster to find out more about root cause.
annunarcist
- 1,637
- 3
- 20
- 42
-
so when a node is lost will the HDFS data in that node also gets lost ? – Sundeep Pidugu Nov 13 '18 at 09:32
-
Yes - if your HDFS replication factor is just 1. No - if its greater than 1 & yours is a multi-node cluster. – annunarcist Nov 22 '18 at 18:31