1

I am trying to submit a spark job through Apache Livy but the LivyClient's uploadJar method is failing.

This is the code (very similar to the PiJob example):

        LivyClientBuilder builder = new LivyClientBuilder();
        LivyClient client = builder.setURI(new URI("http://server:8998")).build();
        client.uploadJar(new File("/path/to/file")).get();

Here is the full stacktrace:

py4j.Py4JException: Error while obtaining a new communication channel
    at py4j.CallbackClient.getConnectionLock(CallbackClient.java:257)
    at py4j.CallbackClient.sendCommand(CallbackClient.java:377)
    at py4j.CallbackClient.sendCommand(CallbackClient.java:356)
    at py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:106)
    at com.sun.proxy.$Proxy24.getLocalTmpDirPath(Unknown Source)
    at org.apache.livy.repl.PythonInterpreter.addPyFile(PythonInterpreter.scala:294)
    at org.apache.livy.repl.ReplDriver$$anonfun$addJarOrPyFile$1.apply(ReplDriver.scala:114)
    at org.apache.livy.repl.ReplDriver$$anonfun$addJarOrPyFile$1.apply(ReplDriver.scala:114)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.livy.repl.ReplDriver.addJarOrPyFile(ReplDriver.scala:114)
    at org.apache.livy.rsc.driver.JobContextImpl.addJarOrPyFile(JobContextImpl.java:151)
    at org.apache.livy.rsc.driver.AddJarJob.call(AddJarJob.java:39)
    at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
    at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:244)
    at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:277)
    at py4j.CallbackConnection.start(CallbackConnection.java:226)
    at py4j.CallbackClient.getConnection(CallbackClient.java:238)
    at py4j.CallbackClient.getConnectionLock(CallbackClient.java:250)
    ... 17 more

I can submit code snippets to the Livy server through the REST API and they run just fine. Livy/Spark is set up on YARN and I have tried both client and cluster mode. Any ideas?

bsannicolas
  • 11
  • 1

1 Answers1

0

From your code it seems that your jar path isn't set up, try to set the path for your jar.

Example

client.uploadJar(new File("tmp/myjar.jar")).get();

You can also check your env path for pyspark environment, be sure that nodes communicates in the same environment.

Shayki Abramczyk
  • 36,824
  • 16
  • 89
  • 114
S. Vladislav
  • 42
  • 6