0

I am working on the following versions:

OS- Windows 8

Hadoop- 2.3 (Single node cluster), Pig- 0.12.1, Hive- 0.13

I could start Namenode, Datanode, ResourceManager and NodeManager using windows command prompt (without admin privileges). When I try to run mapreduce program, Mapping task is not start, It show always as "Mapping-0%".

note: If I start 'NodeManager' from windows command prompt with Administrator privileges then MapReduce program works fine.

NodeManager Log:

14/10/07 14:44:18 INFO http.HttpServer2: adding path spec: /node/*
14/10/07 14:44:18 INFO http.HttpServer2: adding path spec: /ws/*
14/10/07 14:44:18 INFO http.HttpServer2: Jetty bound to port 8042
14/10/07 14:44:18 INFO mortbay.log: jetty-6.1.26
14/10/07 14:44:18 INFO mortbay.log: Extract jar:file:/C:/kaveen/BDSDK/1
.1.0.8/SDK/Hadoop/share/hadoop/yarn/hadoop-yarn-common-2.3.0.jar!/webapps/node t
o C:\Users\kaveen\AppData\Local\Temp\Jetty_localhost_8042_node____j463i9\webapp
14/10/07 14:44:19 INFO mortbay.log: Started SelectChannelConnector@localhost:804
2
14/10/07 14:44:19 INFO webapp.WebApps: Web app /node started at 8042
14/10/07 14:44:19 INFO webapp.WebApps: Registered webapp guice modules
14/10/07 14:44:19 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0
:8031
14/10/07 14:44:19 INFO nodemanager.NodeStatusUpdaterImpl: Registering with RM us
ing finished containers :[]
14/10/07 14:44:19 INFO security.NMContainerTokenSecretManager: Rolling master-ke
y for container-tokens, got key with id 1336526610
14/10/07 14:44:19 INFO security.NMTokenSecretManagerInNM: Rolling master-key for
 nm-tokens, got key with id :-93940432
14/10/07 14:44:19 INFO nodemanager.NodeStatusUpdaterImpl: Registered with Resour
ceManager as 127.0.0.1:49344 with total resource of <memory:8192, vCores:8>
14/10/07 14:44:19 INFO nodemanager.NodeStatusUpdaterImpl: Notifying ContainerMan
ager to unblock new container-requests

ResourceManager Log:

14/10/07 14:44:19 INFO util.RackResolver: Resolved 127.0.0.1 to /default-rack
14/10/07 14:44:19 INFO resourcemanager.ResourceTrackerService: NodeManager from
node 127.0.0.1(cmPort: 49344 httpPort: 8042) registered with capability: <memory
:8192, vCores:8>, assigned nodeId 127.0.0.1:49344
14/10/07 14:44:19 INFO rmnode.RMNodeImpl: 127.0.0.1:49344 Node Transitioned from
 NEW to RUNNING
14/10/07 14:44:19 INFO capacity.CapacityScheduler: Added node 127.0.0.1:49344 cl
usterResource: <memory:16384, vCores:16>
14/10/07 14:44:19 INFO rmnode.RMNodeImpl: Node 127.0.0.1:49344 reported UNHEALTH
Y with details: 1/1 local-dirs turned bad: /tmp/hadoop-kaveen/nm-local-dir;1/1 l
og-dirs turned bad: C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/logs/userlogs
14/10/07 14:44:19 INFO rmnode.RMNodeImpl: 127.0.0.1:49344 Node Transitioned from
 RUNNING to UNHEALTHY
14/10/07 14:44:19 INFO capacity.CapacityScheduler: Removed node 127.0.0.1:49344
clusterResource: <memory:8192, vCores:8>

At cluster Node "http://localhost:8088/cluster/nodes" Look like below sceenshots.

No Nodes are created.

enter image description here

Do anyone know how to run 'NodeManager' without Admin privileges on windows?

Thanks in Advance...

ǨÅVËĔŊ RĀǞĴĄŅ
  • 530
  • 6
  • 30

2 Answers2

1

A quick fix would be to run C:hadoop\sbin\start-dfs and C:hadoop\sbin\start-yarn under Administrator privileges.

Peeter Kokk
  • 1,625
  • 2
  • 15
  • 9
0
local-dirs turned bad: /tmp/hadoop-kaveen/nm-local-dir;1/1 
log-dirs turned bad: C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/logs/userlogs

Your NM does not have a valid local-dirs, you have left Linux style local-dirs that are obviously not going to work on Windows. log-dirs are also incorrect, NM probably does not have the required privileges on it.

There is absolutely no need to elevate NM. Instead, fix local-dirs in yarn-site.xml:

<property>
 <name>yarn.nodemanager.local-dirs</name>
 <value>C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/local</value>
</property>

Then make sure the dirs exists and grant full control to NM:

c:\>md C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\local
c:\>md C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\logs
c:\>cacls  C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\local /E /T /G <nm account>:F
c:\>cacls  C:\kaveen\BDSDK\1.1.0.8\SDK\Hadoop\logs/E /T /G <nm account>:F

Substitute with the appropriate account.

ǨÅVËĔŊ RĀǞĴĄŅ
  • 530
  • 6
  • 30
Remus Rusanu
  • 288,378
  • 40
  • 442
  • 569
  • Hi Remus Rusanu, Thanks for your reply. I changed the step above, but I get same error. "log-dirs turned bad: C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/logs/userlogs" . If I start NodeManager in Admin privilege mean above error was not come. And also I set full permission to "C:/kaveen/BDSDK/1.1.0.8/SDK/Hadoop/logs". – ǨÅVËĔŊ RĀǞĴĄŅ Oct 07 '14 at 10:25
  • You have all the information you need to troubleshoot the problem. – Remus Rusanu Oct 07 '14 at 12:07
  • Thanks Remus Rusanu, I found the solution It due to privilege is not available for my temp folders. – ǨÅVËĔŊ RĀǞĴĄŅ Oct 08 '14 at 05:09
  • @RemusRusanu , any ideas for a similar issue on ubuntu ? http://stackoverflow.com/questions/28949872/haddop-in-lxc-container-error-yarn-1-1-local-dirs-are-bad – AlexandruC Mar 09 '15 at 20:16