Livy create session dead

Question

I added to my spark config a package (in spark-default.conf) but when I create a new session with livy it causes me a problem (see the error below) and the session and death . ps: when I remove this package all work fine .

20/05/04 00:17:35 WARN RSCClient: Error stopping RPC.                                                                                                                   
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@6d493840(uncancellable)                                                                      
        at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:394)                                                                               
        at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)                                                                         
   ...........
Exception in thread "Thread-32" java.io.IOException: Stream closed                                                                                                      
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)                                                                                       
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)                                                                                              
        ........                                                                                            
        at org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:46)                                                                            
20/05/04 00:17:36 WARN ContextLauncher: Child process exited with code 143.                                                                                             
20/05/04 00:17:36 ERROR SparkProcApp: job was killed by user                                                                                                            
20/05/04 00:17:36 INFO InteractiveSession: Stopped InteractiveSession 0.                                                                                                
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleting InteractiveSession 0 because it was inactive for more than 3600000.0 ms.                                     
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleting session 0                                                                                                    
20/05/04 00:28:17 INFO InteractiveSession: Stopping InteractiveSession 0...                                                                                             
20/05/04 00:28:17 INFO InteractiveSession: Stopped InteractiveSession 0.                                                                                                
20/05/04 00:28:17 INFO InteractiveSessionManager: Deleted session 0

I use : cloudera hdp2.6.5 :

spark 2.3
livy 0.7.0
Hadoop 2.7
lib unsupervised (https://github.com/unsupervise/spark-tss)

step :

livy conf => livy.spark.master yarn-cluster
spark-default conf => spark.jars.repositories https://dl.bintray.com/unsupervise/maven/
spark-defaultconf => spark.jars.packages com.github.unsupervise:spark-tss:0.1.1

Could you please provide more details? Which package have you added and how? — Aliaksandr Sasnouskikh, May 10 '20 at 19:45
the package is for lib unsupervised (https://github.com/unsupervise/spark-tss) by adding this line in spark-default : spark.jars.packages com.github.unsupervise:spark-tss:0.1.1 — zekri sidi mohamed hicham, May 11 '20 at 03:32
Depending on your environment they can be stored and accessed in different ways. Please provide more details about your environment. — Aliaksandr Sasnouskikh, May 22 '20 at 20:45

score 0 · Answer 1 · answered May 13 '20 at 20:43

0

Please try also to add to spark defaults

spark.jars.repositories https://dl.bintray.com/unsupervise/maven/

Alternatively (just to be sure that you have no issues with spark-default.conf file) please try to include these configs to the Livy request body instead when submitting the Spark job (refer Livy API).

answered May 13 '20 at 20:43

Aliaksandr Sasnouskikh

921
6
14

I add it. the same issue . This my conf : **spark.jars.repositories https://dl.bintray.com/unsupervise/maven** **spark.jars.packages com.github.unsupervise:spark-tss_2.11:0.1.1,org.mongodb.spark:mongo-spark-connector_2.11:2.4.1** – zekri sidi mohamed hicham May 18 '20 at 12:47
Can you print your Spark (or find it on UI) just to be sure that the values you expect to get are not overwritten? – Aliaksandr Sasnouskikh May 19 '20 at 09:43
the problem is that livy can not load my lib (unsupervised) to the session – zekri sidi mohamed hicham May 19 '20 at 13:56
any help here please – zekri sidi mohamed hicham May 20 '20 at 04:22
Please describe the steps you do so that the issue could be reproduced including the details about the environment. Also please include the versions of the components you run. – Aliaksandr Sasnouskikh May 22 '20 at 20:44
I add it in question – zekri sidi mohamed hicham May 28 '20 at 20:32
Where do you store `livy.conf` and `spark-default.conf` files? Can you be sure that these files are on the classpath of spark-submit launched by Livy? – Aliaksandr Sasnouskikh May 28 '20 at 22:01
Also not sure if `livy.jars.packages` config option works as you expect it, probably you need `spark.jars.packages`. Also the logs you posted are a bit confusing: there are no any word about the added jar. Might be somewhere below or above you can find some related to the package downloading errors!? – Aliaksandr Sasnouskikh May 28 '20 at 22:03
sorry i mean spark.jars.packages not livy.jars.packages . when i remove the lib unsupervise all work well . – zekri sidi mohamed hicham May 29 '20 at 00:17

Livy create session dead

1 Answers1