Failing to submit spark job through Livy programmatic client api

Question

I'm trying to use submit simple word count program which was implemented with java8, spark 2.4.7, and submitted through Livy 0.7. I have three containers which runs on my local computer. One for Spark driver, one for Spark worker and one for Livy.

I use the Livy's client to submit the program as follows:

private static final String LIVY_URL = "http://localhost:8998/";
private static final String JAR_PATH = "/.../spark-word-count/target/word-count-0.1- 
SNAPSHOT.jar";

public static void main(String[] args) throws Exception {
    LivyClient livyClient = new LivyClientBuilder(false)
    .setURI(new URI(LIVY_URL)).build();
    
    try {
        System.err.printf("Uploading %s to the Spark context...\n", JAR_PATH);
        livyClient.uploadJar(new File(JAR_PATH)).get();
        
        System.err.printf("Running JavaWordCount...\n", JAR_PATH);
        double returnValue = livyClient.submit(new JavaWordCount()).get();
      
        System.out.println("Application return value is: " + returnValue);
      } finally {
        livyClient.stop(true);
      }
}

And I'm getting the following error:

Caused by: java.io.IOException: Bad Request: {"msg":"requirement failed: Local path /root/.livy-sessions/b78e9935-70b4-4fe7-8840-8c84067fbd52/word-count-0.1-SNAPSHOT.jar cannot be added to user sessions."}

Looking at Livy's UI, I can see that a session starts, and when I attach to Livy's running container I can see my Jar uploaded to /root/.livy-sessions, but then it fails with the error above.

The docker-compose file that I use pointing LIVY_FILE_LOCAL_DIR_WHITELIST to /opt/jars. I saw that people suggest to change it to the location where the Jar is uploaded to, I indeed change it to /root/.livy-sessions, but I'm getting some other error:

Exception in thread "main" java.util.concurrent.ExecutionException: java.io.IOException: Internal Server Error: "java.util.concurrent.ExecutionException: java.lang.RuntimeException: py4j.Py4JException: Error while obtaining a new communication channel

I will really appreciate any help! thanks...

Failing to submit spark job through Livy programmatic client api

0 Answers0