Run my first code of pyspark in linux ubuntu 18.04.5

Question

I've learned on how to setup pyspark on multi-node with much of googling. And now i've created my first pyspark code with only creating a dataframe and printing its data inside of it.

But now I wanted to run and execute my pyspark code, i named it "firstcode.py". I've tried to search but didn't get a clue how to do that. Like how to run it, and where should i execute the script to run my "firstcode.py" files?

https://stackoverflow.com/questions/38120011/using-spark-submit-with-python-main — Compsci is math base 2, Dec 15 '22 at 20:21
Please provide enough code so others can better understand or reproduce the problem. — Community, Dec 16 '22 at 10:07

score 0 · Answer 1 · answered Dec 15 '22 at 20:46

You can submit your code using spark submit on any host in your cluster. Resource usage, additional libraries vs can be configured. master url can be vary as yarn, local,standalone, kubernetes etc. You can see submitting application for details.

Sample script for master as yarn:

spark-submit \
 --master yarn \
 --deploy-mode cluster \
 --executor-memory 1g\
 --num-executors 2\
 myCode.py

Sample script for standalone spark

spark-submit \
--master spark://host_ip:7077 \
--deploy-mode cluster \
--executor-memory 1g \
myCode.py

Run my first code of pyspark in linux ubuntu 18.04.5

1 Answers1