0

"Internal Server Error: Session is in state starting" error while running sample Spark Livy job.

I installed Livy for accessing apache-spark using REST API and started the Livy server. On running a sample spark Pi App(as giving on https://github.com/cloudera/livy ), I am getting error message "Caused by: java.io.IOException: Internal Server Error: "java.lang.IllegalStateException: Session is in state starting".

Exception in thread "main" java.util.concurrent.ExecutionException: java.io.IOException: Internal Server Error: "java.lang.IllegalStateException: Session is in state starting" at com.cloudera.livy.client.http.JobHandleImpl.get(JobHandleImpl.java:201) at com.cloudera.livy.client.http.JobHandleImpl.get(JobHandleImpl.java:91) at PiApp.main(PiApp.java:96) Caused by: java.io.IOException: Internal Server Error: "java.lang.IllegalStateException: Session is in state starting" at com.cloudera.livy.client.http.LivyConnection.sendRequest(LivyConnection.java:197) at com.cloudera.livy.client.http.LivyConnection.sendJSONRequest(LivyConnection.java:173) at com.cloudera.livy.client.http.LivyConnection.post(LivyConnection.java:149) at com.cloudera.livy.client.http.JobHandleImpl$1.run(JobHandleImpl.java:145) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Saurabh Rana
  • 168
  • 3
  • 22

1 Answers1

0

Even I got the same error while running the PiApp.java from my Eclipse and connecting to remote Livy server. But when I ran the very same PiApp jar directly at the remote server where Livy and Spark are installed then it ran fine...

java -cp /home/Anshul/livy/spark-core_2.10-1.6.0.jar:/home/Anshul/livy/livy-api-0.3.0.jar:/home/Anshul/livy/livy-client-http-0.3.0.jar:/home/Anshul/livy/LivyApp-0.0.1-SNAPSHOT.jar com.cloudera.livy.examples.PiApp http://sandbox.hortonworks.com:8998 2

Anshul Gupta
  • 490
  • 4
  • 8
  • Thanks. Are we not supposed to run it using spark-submit? I have been trying to run it using SparkSubmit. – Saurabh Rana Mar 22 '17 at 14:12
  • I tried this command :java -cp LivyClient1-jar-with-dependencies.jar:/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p1464.1349/jars/spark-core_2.10-1.6.0-cdh5.7.0.jar:/home/ubuntu/softwares/livy-server-0.2.0/jars/livy-api-0.2.0.jar:/home/ubuntu/softwares/livy-server-0.2.0/jars/livy-client-common-0.2.0.jar PiApp http://hostname:8998 2 . However it gave same error: Caused by: java.io.IOException: Internal Server Error: "java.lang.IllegalStateException: Session is in state starting" com.cloudera.livy.client.http.LivyConnection.sendRequest(LivyConnection.java:197) – Saurabh Rana Mar 22 '17 at 14:53
  • Spark-Submit is definitely not applicable here, as main responsibility of Livy Server is to avoid our client applications bother about spark context and submits... As for your error, please try with livy 0.3.0 jars which worked for me.... Although technically I am not able to solve this issue when I am running directly from eclipse. I have converted my Livy App into a Spring Boot application, still this error is coming... Only way I am able to successfully run it is using livy jar 0.3.0 externally like java -cp – Anshul Gupta Mar 24 '17 at 06:05
  • Thanks. will change it to version 0.3.0 and try. – Saurabh Rana Mar 27 '17 at 12:58
  • Thanks . Using the version 0.3.0 I do not get that error. Now I face another issue that submitting the spark job using Livy Rest request makes the job go in dead state.{"id":12,"state":"dead","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null} – Saurabh Rana Mar 29 '17 at 13:16
  • Couple of new configurations which I had found are below....... livy_server_session_kind=yarn livy.file.local-dir-whitelist=/home/livy/.livy-sessions Also surprisingly LivyApp only works by using livy-client-http jar from cloudera and not the livy-api jar in apache incubating project....... com.cloudera.livy livy-client-http 0.3.0 – Anshul Gupta Nov 30 '17 at 05:31