1

I've had a series of devops issues from time to time on our production cluster. Every now and then, / partition gets overwhelmed on couple of nodes. Long story short, it turns out that these nodes had 1 instead of 2 data drives. This would not be an issue if we don't have a following setup on our cluster:

   <property>       
      <name>yarn.nodemanager.local-dirs</name>       
      <value>/data1/hadoop/yarn/local,/data2/hadoop/yarn/local</value> 
   </property>

Some devops or whoever, noticing there are no /data2 partitions on the smaller nodes, came up with the idea to simply go with / partition. Since / is 16GB, some of the more data-demanding jobs quickly fill the thing.

Now, my question: does yarn support per-node setup of yarn.nodemanager.local-dirs?

I resolved the problem by removing /data2/hadoop/yarn/local from the story, but it doesn't feel perfect.

We're using HDP 2.6.4.

Thx!

hummingBird
  • 2,495
  • 3
  • 23
  • 43

1 Answers1

2

YARN allows this since each Node Manager would read it's local yarn-site.xml. However, I don't know how you would do this in Ambari.

tk421
  • 5,775
  • 6
  • 23
  • 34