Livy Batch API - not able to get the jar

Question

I am trying to use Livy Batch API to invoke my Spark application.

curl -X POST -d '{"file": "file:///path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar","className": "App","args": ["abc"],"jars": [],"driverMemory": "16g","executorMemory": "8g","executorCores": 2,"numExecutors": 2,"queue": "default","name": "Livy_sample_prog"}' -H "Content-Type: application/json" http://serverip:8998/batches/

POST/batches is failing with the below error:

"Warning: Local jar /path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar does not exist, skipping.",
  "java.lang.ClassNotFoundException: App",

Below are the various things I tried so far

1.Uploaded the jar to HDFS

curl -X POST -d '{"file": "hdfs://path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar","className": "App","args": ["abc"],"jars": [],"driverMemory": "16g","executorMemory": "8g","executorCores": 2,"numExecutors": 2,"queue": "default","name": "Livy_sample_prog"}' -H "Content-Type: application/json" http://serverip:8998/batches/

exception:

"Warning: Skip remote jar hdfs://path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar.",
  "java.lang.ClassNotFoundException: App"

2.added livy.file.local-dir-whitelist as dir which contains the jar file

3.changed file:/// to local:/

I have verified several times the files is present and the path provided in each case is valid.

I am out of options now. Please help to find a solution.

Thanks!!

Use Scala, make your life far easier. – thebluephantom Apr 04 '20 at 10:34 — thebluephantom, Apr 04 '20 at 10:34

score 0 · Accepted Answer · answered Apr 07 '20 at 16:32

I was able to make this work my giving master as yarn-cluster insted of yarn.

Here is the --data I used

{
    "file": "local:/path/to/file/SampleSparkProject-0.0.2-SNAPSHOT.jar",
    "proxyUser": "nrsapp",
    "className": "App",
    "args": [
        "abc"
    ],
    "name": "Livy_sample_prog",
    "conf": {
        "spark.master": "yarn-cluster",
        "spark.deployMode": "client"
    }
}

Livy Batch API - not able to get the jar

1 Answers1