1

I am trying to execute spark submit job with python3 and python2 compiled mypthon3.zip on Databricks DRE 7.6 and above and getting below error

Traceback (most recent call last):
  File "/dbfs/tmp/WT_SPARK3/Inputsql.py", line 1, in <module>
    from com.test.mymodule import Mymodule
zipimport.ZipImportError: can't find module 'com'

mypython3.zip : compiled with python3 and contains pyc file with pycache folder

command to compile : python3 -m compileall -b package_dir

mypython3.zip
  |- com
      |- __pycache__ (empty folder)
      |- __init__.pyc
      |- test
           |- __pycache__ (empty folder)
           |- __init__.pyc
           |- mymodule.pyc

Inputsql.py

from com.test.mymodule import Mymodule
import sys
import time
import logging

spark submit command

["--py-files","/dbfs/tmp/mypthon3.zip","/dbfs/tmp/Inputsql.py"]

Please let me know how to fix it

Thanks

Utkarsh Jain
  • 39
  • 1
  • 4

0 Answers0