What should the file permissions be for yarn.nodemanager.local-dirs folders? I have messed up the file permissions for these and not able to solve it. Currently I have set the folder permissions to be 755 yarn:hadoop. Whenever I try to run the nodemanager, I run into EPERM: Operation not permitted error. Details -
2014-06-19 22:15:17,954 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: usercache path : file://opt1/yarn/nm2/usercache_DEL_1403241103430
2014-06-19 22:15:17,954 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: usercache path : file://opt1/yarn/nm2/usercache_DEL_1403241317919
2014-06-19 22:15:17,954 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: usercache path : file://opt1/yarn/nm2/usercache_DEL_1403241109852
2014-06-19 22:15:17,954 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: usercache path : file://opt1/yarn/nm2/usercache_DEL_1403241098628
2014-06-19 22:15:17,992 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize LocalizationService
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize LocalizationService
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceInit(ResourceLocalizationService.java:233)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:220)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:188)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:353)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:401)
Caused by: EPERM: Operation not permitted
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:226)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:629)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:421)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1061)
at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:157)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:720)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:716)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:716)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceInit(ResourceLocalizationService.java:230)
... 9 more
The local-dirs folder has the following folders after the failed start attempt - filecache, nmPrivate, 3 usercache_DEL_timestamp. Based on the location of the error(ResourceLocalizationService.java, line 233), three possible actions could raise this error -
a) Cleaning up old files - this works, as reflected in log output
b) Creating new files - After the failed start attempt, I see three kinds of files in local-dirs - three usercache_DEL_*, filecache, nmPrivate files.
c) Making log paths - I have checked the log folders to have permission 775 for yarn:hadoop.
What is possibly going wrong? I'm using Cloudera to manager my cluster.
