I am trying to execute spark submit job with python3 and python2 compiled mypthon3.zip on Databricks DRE 7.6 and above and getting below error
Traceback (most recent call last):
File "/dbfs/tmp/WT_SPARK3/Inputsql.py", line 1, in <module>
from com.test.mymodule import Mymodule
zipimport.ZipImportError: can't find module 'com'
mypython3.zip : compiled with python3 and contains pyc file with pycache folder
command to compile : python3 -m compileall -b package_dir
mypython3.zip
|- com
|- __pycache__ (empty folder)
|- __init__.pyc
|- test
|- __pycache__ (empty folder)
|- __init__.pyc
|- mymodule.pyc
Inputsql.py
from com.test.mymodule import Mymodule
import sys
import time
import logging
spark submit command
["--py-files","/dbfs/tmp/mypthon3.zip","/dbfs/tmp/Inputsql.py"]
Please let me know how to fix it
Thanks