execute batch job on livy within python virtualenv created already in server

Question

I am trying to execute a python file using batch job on livy

Url: http://<server-host>:8998/batches

body:

{
    "file": "hdfs://localhost:9000/sample.py"
}

i want to execute this file within python virtualenv which is already created on server. I am new to livy, can someone please help me with this?

some references for you - https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/running-spark-applications/content/livy_api_reference_for_batch_jobs.html https://gethue.com/how-to-use-the-livy-spark-rest-job-server-api-for-submitting-batch-jar-python-and-streaming-spark-jobs/ — dsk, Jun 15 '20 at 10:00

score 0 · Answer 1 · answered Jun 15 '20 at 11:30

0

adding the below configuration of python(virtual environment) to be used solved my problem

{
    "file": "hdfs://localhost:9000/sample.py",
    "conf": {
        "spark.pyspark.python": "/opt/project/project_env/bin/python"
     }
}

Thank you.

answered Jun 15 '20 at 11:30

Akshata Dasar

1 Answers1