0

I'm trying to create spark interactive session with livy .and I need to add a lib like a jar that I mi in the hdfs (see my code ) . but the session is dead and the log is below.

code :

client = LivyClient('http://sandbox.c4e.kyomei.fr:10500')
session = client.create_session(SessionKind.SPARK , jars = ['hdfs://sandbox-hdp.hortonworks.com:8020/tmp/tsa2-assembly-0.1.jar'] )

log : 

> 20/05/09 01:43:48 INFO LineBufferedStream: Exception in thread "main"
> scala.reflect.internal.FatalError: object Predef does not have a
> member classOf                    20/05/09 01:43:48 WARN RSCClient:
> Error stopping RPC.                                                   
> io.netty.util.concurrent.BlockingOperationException:
> DefaultChannelPromise@786d2cd8(uncancellable)                         
> 
>         at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:394)
> 
>         at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
> 
>         at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:230)
> ....

> Exception in thread "Thread-32" java.io.IOException: Stream closed    
> 
>         at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
> 
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)       
> 
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:345)

> 20/05/09 01:43:48 WARN ContextLauncher: Child process exited with code
20/05/09 01:43:48 ERROR SparkProcApp: job was killed by user
zekri sidi mohamed hicham
  • 57
  • 1
  • 10
  • Can you give more info about the code and setup? What Livy version are you using? What Livy client are you using? You don't seem to be using the `livy-client-http` library. Also, does the spark code work when you run it directly through spark-submit (instead of going through livy)? – user1516867 May 09 '20 at 06:20
  • I use pylivy for lib (the same error when i use livy-client-http) . I use livy 0.7.0 with spark 2.4.5 . all work well when i use spark-shell with option --jars – zekri sidi mohamed hicham May 09 '20 at 09:55

0 Answers0