0

I am using livy(post/batches) to submit a jar with keyword parameters.

For example:

spark-sumbit \
    --class xxx \
    --master xxx \
    --conf xxx=aa \
    my_test.jar --arg1 <arg1> --arg2 <arg2>

In livy(post/batches), how can I do this? Does livy support this?

allinone
  • 1
  • 2

2 Answers2

0

You can follow this format wjile POST

{ "file": "xxx.jar", "className": "xx", "args": ["xx"], "conf": { "spark.jars.packages": "com.amazonaws:aws-java-sdk-bundle:1.11.199,org.apache.hadoop:hadoop-aws:2.9.1, "spark.hadoop.fs.s3a.signing-algorithm": "S3SignerType", "spark.ui.port":"4040", "spark.ui.proxyBase":"/hisparklivy", "spark.driver.memory": "4g", "spark.executor.memory": "8g", "spark.executor.cores": 5, "spark.dynamicAllocation.enabled": "false", "spark.cores.max": 25}

kushal
  • 1
  • 1
  • Thank you for you answer! Maybe I didn't describe my problem clearly. In the old version of my_test.jar, I use location parameters. It worked correctly after I POST `{..., "args": ["val1", "val2"],...}`. But in the new version, I use keyword parameters, and POST `{..., "args": ["--args1 val1", "--args2 val2"], ...}`. It turned out to be wrong. – allinone Jul 26 '19 at 13:39
  • 2
    After some trial and error, I found the right way. I should post `{..., "args": ["--args1", "val1", "--args2", "val2"], ...}` , not `{..., "args": ["--args1 val1", "--args2 val2"], ...}` – allinone Jul 26 '19 at 13:43
0

I've had an experience with livy that led me here to stackoverflow , what you need is to call your jars,in the curl command as explained here in this answer.

If you're using a local jar, think about overriding the "livy.file.local-dir-whitelist" in the livy confuration file.

Ahmed Adnane A'mil
  • 25
  • 1
  • 10