1

I am trying to submit a spark job through livy. In the job , I need to post a json data in the args parameter while invoking livy.This is what I have done

String payload = "{\"name\": \"myname\", \"id\": \"101\"}";
String data="{ \"file\":\"hdfs://some.jar\" ,\"className\":\"someclass\",\"args\":["+"\""+payload+"\""+"], \"proxyUser\":\"clsadmin\"}";
String[] command = {"curl", "-v","-g", "-u" ,"id:pwd", "-H", "Content-Type:application/json" ,  "-H", "X-Requested-By:livy" , url ,"--data", data};

It does not work. Now if I change my payload to this.It works.

String payload = "{\\\"name\\\": \\\"myname\\\", \\\"id\\\": \\\"101\\\"}";

How can I avoid using 3 backslashes in the payload ? Is there a better way to do it. I am using curl to call the livy url.

Process p = process.start();
BufferedReader reader =  new BufferedReader(new InputStreamReader(p.getInputStream()));
StringBuilder builder = new StringBuilder();
String line = null;
while ( (line = reader.readLine()) != null) 
{
    builder.append(line);
    builder.append(System.getProperty("line.separator"));
}
Ayan Biswas
  • 1,641
  • 9
  • 39
  • 66

1 Answers1

0

In Livy rest API, args is command line arguments for the application. It is basically and array of strings but in your scenario, you are passing an array of json. Below is an example of your spark-submit command and your args parameter in rest API call.

spark-submit --class className hdfs://some.jar myname 101

args : [ "myname", "101" ]
Apurba Pandey
  • 1,061
  • 10
  • 21
  • but in this how would I mention the name of the parameter ? as using this command i can only send the value of the param in args ?also for me the command line argument is a json data.how do I send that ? – Ayan Biswas Mar 05 '19 at 08:32
  • Json will always have spaces in the data, and command line arguments are space separated. are you able to send json as command line arguments in your spark-submit? Alternatively you can use colon or "=" separated values in command line and read it the same way in your class. For example, name=myname. – Apurba Pandey Mar 05 '19 at 09:01
  • yes through direct spark-submit I am able to add the json as a command line arg. like this :spark-submit --class someclass --master yarn somejar.jar '{"name":"ayan" , "id":"101"}' – Ayan Biswas Mar 05 '19 at 09:14
  • Then you have to pass the entire json as string because args will accept array of strings only. – Apurba Pandey Mar 05 '19 at 09:56