0

I'm trying to launch hello-samza example starting from master branch. I've run every command without errors, started run-job.sh without errors, but job in YARN stays forever in ACCEPTED state.

I've looked at http://localhost:8088/cluster/nodes and it shows nothing - is this the problem? YARN has no nodes connected so it cannot allocate resources to complete submitted job?

yarn node -list

also shows Total Nodes:0

Why is that, if I'm simply following the instructions?

Jon Bringhurst
  • 1,340
  • 1
  • 10
  • 21
grz.miejski
  • 173
  • 1
  • 2
  • 10

2 Answers2

0

YARN has no nodes connected so it cannot allocate resources to complete submitted job?

Yes, the job is queued up and is awaiting resources to launch.

It may be useful to look at the logs for the NodeManager to figure out why it can't connect to the ResourceManager.

Jon Bringhurst
  • 1,340
  • 1
  • 10
  • 21
0

I had the same problem but I figured it out what was wrong. Maybe in your situation it was different problem, but it is worth to check it (also it may help other people). According to suggestion of Jon Bringhurst I recommended to look at Yarn Node Manager logs - for hello samza project they should be under: 

hello-samza/deploy/yarn/logs/yarn-*nodemanger-*.log

In my case (probably in your too) I saw ERORR which said (before there was also WARN which tell me that disk space is above 90%):

1/1 local-dirs are bad: *; 
1/1 log-dirs are bad: *

They are bad because available disk space on the node exceeding yarn's max-disk-utilization-per-disk-percentage default value of 90.0%. Either clean up the disk, or increase the threshold in yarn-site.xml (like I did):

<property>
        <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
        <value>97</value>
</property>
tmucha
  • 689
  • 1
  • 4
  • 19