Calling Apache Livy REST API from Java

Question

We use Apache Livy to read data from object storage in a Spark cluster.

Apache Livy API: POST http://server:port/sessions/{sessionId}/statements Request Payload:

{"code":" val df = spark.read.option(\"header\", true).json(\"xxx://somebucket@somenamespace/xyz\"); \n    df.toJSON.collect.mkString(\"[\", \",\", \"]\")"}

I see the above working with postman without issues. We need to call this API from Java. When we tried to call the above API from Java App using a Jersey Client(version 2.30.1 on Java 11), Apache Livy server returns 'Internal Server Error' (500). The Jersey client code looks like as shown below:


JSONObject payLoad = new JSONObject();
String codeBlock = "val df = spark.read.option(\"header\", true).json(\"xxx://somebucket@somenamespace/xyz\"); \n    df.toJSON.collect.mkString(\"[\", \",\", \"]\")";
payLoad.put("code", codeBlock);

// In this e.g, session id: '0' is created before calling the below API
String apacheLivyUrl = "http://localhost:8998/sessions/0/statements";
Response response = ClientBuilder.newBuilder().build().target(apacheLivyUrl)
        .request(MediaType.APPLICATION_JSON)
        .accept(MediaType.APPLICATION_JSON)
        .post(Entity.json(payLoad.toString()));

Any idea what is going wrong here? Thanks for your time

are you sure you're using the same URL in both cases? (including the same session ID) — mangusta, Jun 24 '20 at 07:04
by the way have you tried using it without `.accept(MediaType.APPLICATION_JSON)`? — mangusta, Jun 24 '20 at 07:13

score 0 · Accepted Answer · answered Jun 25 '20 at 04:16

It happens in the following scenario: We have a Java API to execute statement and it does the following:

Create a session by calling Livy /sessions REST API
Then create a statement by passing the session id from prev step with Livy REST API POST //statements. In this case, Step 2 gives 500 error code.

By looking at the response for POST /session api call(step 1), noticed a 'state' flag in response body(it was state='starting' when it failed with status 500). Apparently, one need to check the status of the state flag before proceeding with Step 2. There is a GET /sessions/ Levy REST API to check the state status (till it changes from 'starting') to proceed further with statement creation.

Calling Apache Livy REST API from Java

1 Answers1