0

I am trying to run spark job on Yarn cluster in "yarn-client" mode; using Oozie shell action to issue spark-submit command. Note that oozie job is triggered by logged in user ("my-user"), but i get Execute permission issue.Please refer log below

For more detailed output, check application tracking page:http://physrv3:8088/proxy/application_1452252389874_0018/Then, click on links to logs of each attempt.
Diagnostics: Permission denied: user=my-user, access=EXECUTE, inode="/user/yarn/.sparkStaging/application_1452252389874_0018":yarn:hdfs:drwx------
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6609)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4223)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:894)
    at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getFileInfo(AuthorizationProviderProxyClientProtocol.java:526)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:822)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
NSNoob
  • 5,548
  • 6
  • 41
  • 54
Manohar CS
  • 31
  • 8
  • Please edit your question with `core-site.xml` and `yarn-site.xml` contents. Have you added `hadoop.proxyuser.[OOZIE_SERVER_USER].hosts` to your `core-site.xml` ? – Mobin Ranjbar Jan 11 '16 at 09:15
  • Is this what you are referring to Mobin? hadoop.proxyuser.oozie.hosts * hadoop.proxyuser.oozie.groups * – Manohar CS Jan 11 '16 at 09:27
  • We need to refer to this portion from log
    `code` Diagnostics: Permission denied: user=my-user, access=EXECUTE, inode="/user/yarn/.sparkStaging/application_1452252389874_0018":yarn:hdfs:drwx------ at `code`     – Manohar CS Jan 11 '16 at 09:32
  • Yes, change it to `hadoop.proxyuser.my-user.hosts` and `hadoop.proxyuser.my-user.groups` – Mobin Ranjbar Jan 11 '16 at 09:41
  • Thanks for quick response. i will make changes accordingly and verify if it helps. – Manohar CS Jan 11 '16 at 09:46
  • Hi Mobin, I tried adding hadoop.proxyuser.my-user.hosts and hadoop.proxyuser.my-user.groups to core-site.xml but still gets the same error `Permission denied: user=my-user, access=EXECUTE, inode="/user/yarn/.sparkStaging/application_1453368435324_0026":yarn:hadoop:drwx------` – Manohar CS Jan 21 '16 at 10:25
  • You do not have access to /user/yarn. You have to put your staging directory in `/user/my-user` or change the owner of `/user/yarn` by `hdfs dfs -chown my-user:my-user /user/yarn -R` – Mobin Ranjbar Jan 21 '16 at 12:15
  • I had a similar problem today. The issue with shell-action is that you cannot change the "yarn" user even if you sudo in the shell script! Quoting "Command can't be executed as different user using sudo." from http://oozie.apache.org/docs/4.1.0/DG_ShellActionExtension.html – Zouzias Jan 26 '16 at 19:56
  • @Zouzias - did u solve this problem? what was the work around tried? – Manohar CS Jan 28 '16 at 11:25
  • I couldn't find a solution using shell-action. However, I managed to run a spark job using a java-action by following http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow – Zouzias Jan 29 '16 at 11:09

1 Answers1

0

I would suggest you to use a java action as described Spark with Java Action if you have Oozie version smaller than 4.2.

Otherwise, if you have admin rights to the oozie server, upgrade to Oozie version >= 4.2.x where there is a native support for Spark actions.

Community
  • 1
  • 1
Zouzias
  • 2,330
  • 1
  • 22
  • 32