I have trouble with using apache-livy, my interpreter doesn't work

Question

I am currently trying to make a REST API server with apache livy. I started a livy-server and with my python code I used the livy api.

Here's my python code that interacts with the livy-server

import json, pprint, requests, textwrap

#start session
host = 'http://localhost:8998'
data = {'kind': 'pyspark'}
headers = {'Content-Type': 'application/json'}
r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
pprint.pprint(r.json())


session_url = host + r.headers['location']
print(r.headers['location'])
r = requests.get(session_url, headers=headers)
pprint.pprint(r.json()['state'])


r = requests.get(session_url, headers=headers)
response = r.json()
while response['state'] == "starting":
    r = requests.get(session_url, headers=headers)
    response = r.json()
r = requests.get(session_url, headers=headers)
pprint.pprint(r.json()['state'])


# now server is ready POST data
statements_url = session_url + '/statements'
data = {
    'code': textwrap.dedent("""
    1+1
    """)
}
r = requests.post(statements_url, data=json.dumps(data), headers=headers)
pprint.pprint(r.json())
request = r.json()
r = requests.get(statements_url, headers=headers)
while r.json()['statements'][0]['state'] == "waiting":
    r = requests.get(statements_url, headers=headers)
    print("statements : ",r.json()['statements'][0]['state'])
r = requests.get(statements_url, headers=headers)
pprint.pprint(r.json())

The livy-server doesn't work properly. When I send data to http://localhost:8998/sessions/0/statements

livy-server raises an "interpreter error".

this is the output of my python code:

{'statements': [{'code': '\n1+1\n',
                 'completed': 1641642497713,
                 'id': 0,
                 'output': {'ename': 'InterpreterError',
                            'evalue': 'Fail to start interpreter',
                            'execution_count': 0,
                            'status': 'error',
                            'traceback': []},
                 'progress': 1.0,
                 'started': 1641642497705,
                 'state': 'available'}],
 'total_statements': 1}

I don't know what's going on.

Do I have to do something for the "interpreter?"

There's no solution for this on the official livy website.

What is the livy/spark route behind the /sessions route? – Braiam Jan 11 '22 at 20:35 — Braiam, Jan 11 '22 at 20:35

score 0 · Answer 1 · answered Jan 19 '22 at 05:07

0

I solved the problem myself.

Scala version was the problem.

answered Jan 19 '22 at 05:07

cicada

19
2

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jan 19 '22 at 12:28

I have trouble with using apache-livy, my interpreter doesn't work

1 Answers1