I'm trying to run spark submit(pyspark) command. As part of the spark submit, I need to provide the dependency of boto3 as it is dependency in my code. I'm running the below command and getting no module error.
bin/spark-submit --master=local --py-files /home/user/boto3-develop.zip /home/user/py_script.py
Traceback (most recent call last):
File "/home/user/py_script.py", line 16, in <module>
import boto3
ModuleNotFoundError: No module named 'boto3'
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
import apport.fileutils
File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
from apport.packaging_impl import impl as packaging
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
import apt
File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'
Original exception was:
Traceback (most recent call last):
File "/home/user/py_script.py", line 16, in <module>
import boto3
ModuleNotFoundError: No module named 'boto3'
Not sure where I'm going wrong.