Zeppelin --> Shiro --> Livy Integration Error: Cannot start spark | livy is not allowed to impersonate user1

Question

I am facing an issue with Zeppelin --> Shiro --> Livy Integration. It would be great if someone could help me on this.

My current environment set up as follows:

• 1 Master node and 2 slave nodes running.
• Zeppelin installed on Master node up and running
• Shiro authentication has been enabled using shiro.ini file and zeppelin with shiro works fine as well(No LDAP authentication Yet)
• Livy server installed on Master Node Up and running

core-site.xml under etc/hadoop has been configured as follows:

  <property>
    <name>hadoop.proxyuser.livy.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.livy.groups</name>
    <value>*</value>
  </property>


*************************************************************
Also livy.conf under /livy/conf has been ocnfigurd as follows:
*************************************************************

# What port to start the server on.
livy.server.port                 8998

# What spark master Livy sessions should use.
livy.spark.master                yarn

livy.impersonation.enabled true


***************************************************************************
From Zeppelin UI , I have configured the %livy interpreter with below values
***************************************************************************

livy.spark.master   : local[*]
zeppelin.livy.url   : http://localhost:8998


 My Testing:
    Logged into to Zeppelin as “user1“ successfully. And to test the connectivity between the Zeppelin  - -> Shiro -- > Livy Integration I am running below simple codes .

Code 1:

%livy.spark
val accum = sc.accumulator(0) 
sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum += x)
accum.value


Output Error: Cannot start  spark.

Code 2:
%livy.spark
sc.version


Output Error on zeppelin ui: --> Cannot start  spark.

Code3:
%livy.pyspark
print("1")



Output Error: --> Cannot start spark.

Issue:
The notebook takes a while for running and throws error message "Cannot start spark." on the zeppelin UI against the notebook output. 
Further while investigating the Log file “livy-livy-server.out” under ,Log File Path: /var/log/livy/  

Below error is visible on the log file.

18/04/09 12:15:25 INFO WebServer: Starting server on http://IReplcaedmyHostNameFromHere:8999
18/04/09 12:17:56 INFO InteractiveSession$: Creating Interactive session 0: [owner: null, request: [kind: spark, proxyUser: Some(user1), conf: spark.master -> local[*], heartbeatTimeoutInSecond: 0]]
18/04/09 12:17:56 INFO RpcServer: Connected to the port 10001
18/04/09 12:17:56 WARN RSCConf: Your hostname, <My Host>, resolves to a loopback address, but we couldn't find any external IP address!
18/04/09 12:17:56 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
18/04/09 12:17:56 INFO InteractiveSessionManager: Registering new session 0
18/04/09 12:17:59 INFO LineBufferedStream: stdout: 18/04/09 12:17:59 INFO RSCDriver: Connecting to: IReplcaedmyHostNameFromHere.internal:10001
.
.
.
.
18/04/09 12:18:07 INFO LineBufferedStream: stdout: ERROR: org.apache.hadoop.security.authorize.AuthorizationException: User: livy is not allowed to impersonate user1

Don't set livy.spark.master to local[*] set it to yarn-cluster — zjffdu, Apr 11 '18 at 07:25
@zjffdu ... I made below changes. 1. Set the livy.spark.master to yarn-cluster 2. Restarted the livy server 3. Restarted zeppelin instance However the error still remains the same. zepplin UI Error : Cannot start spark Error on Livy Log File: Log file name: livy-livy-server.out Error Message: livy is not allowed to impersonate user1 Do you think there is any access permission that i have missed some where ? — Data Analyst, Apr 15 '18 at 11:26
Do you restart hadoop after you apply the core-site changes ? — zjffdu, Apr 15 '18 at 13:50
Will try to restart Hadoop and check the behaviour again. Many thanks for your immediate response. — Data Analyst, Apr 15 '18 at 15:06

Zeppelin --> Shiro --> Livy Integration Error: Cannot start spark | livy is not allowed to impersonate user1

0 Answers0