5

I'm trying to run spark submit(pyspark) command. As part of the spark submit, I need to provide the dependency of boto3 as it is dependency in my code. I'm running the below command and getting no module error.

bin/spark-submit --master=local --py-files /home/user/boto3-develop.zip /home/user/py_script.py

Traceback (most recent call last):
  File "/home/user/py_script.py", line 16, in <module>
    import boto3
ModuleNotFoundError: No module named 'boto3'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "/home/user/py_script.py", line 16, in <module>
    import boto3
ModuleNotFoundError: No module named 'boto3'

Not sure where I'm going wrong.

data_addict
  • 816
  • 3
  • 15
  • 32

0 Answers0