VS Code throws ModuleNotFoundError despite folder available

Question

I'm working on creating a Python/PySpark library using VS Code. My goal is to debug in VS Code and create a .whl package to be installed in a Databricks cluster. I face the following situations:

if I use from checkenginelib.pysparkdq._constraints._Constraint import _Constraint I get a ModuleNotFoundError in VS Code and a module not found error in Databricks
if I use from pysparkdq._constraints._Constraint import _Constraint I get a ModuleNotFoundError in VS Code but all imports work well in Databricks
if I use from _constraints._Constraint import _Constraint I get no error in VS Code but I get a module not found error in Databricks

If the python file is in the same directory it is sufficient to just import the .py file. In this case then `from validate_df import...` — Sabsa, Jul 22 '22 at 07:30

score 1 · Answer 1 · edited Jul 23 '22 at 13:41

1

From what I see, you are working in ./DATA-QUALITY-ENGINE/check-engine-lib/dqengine/validate_df. You have to import it the following way:

from check-engine-lib.dqengine.validate_df import *

That should work. Also you need to create a \__init__.py file to import other files as modules

edited Jul 23 '22 at 13:41

Rabinzel

7,757
3
10
30

answered Jul 22 '22 at 07:30

I get an syntax error when running ```from check-engine-lib.dqengine.validate_df import *``` due to the ```-```, is that normal? – Luiz Viola Jul 22 '22 at 08:12
Yes, it is. Unfortunately you'll have to change the directory's name to check_engine_lib I suspect – Jul 22 '22 at 08:29
I followed your sugestion but still got the same error – Luiz Viola Jul 25 '22 at 13:40
Have you added the `__init__.py` file? – Jul 26 '22 at 12:46
yes, in every folder... – Luiz Viola Jul 26 '22 at 13:23

score 1 · Answer 2 · answered Jul 22 '22 at 07:36

1

Because your module dqengine is not in the top level folder, it is probably not in your PYTHONPATH variable, which VSCode has probably added the path to DATA QUALITY ENGINE

Either:

Move it to the top level folder (Data quality engine)
add the path to check_engine_lib to PYTHONPATH.
Or as @franjefriten says, add an __init__ to check-engine-lib and do

from check-engine-lib.dqengine.validate_df import *

answered Jul 22 '22 at 07:36

Tom McLean

5,583
1
11
36

I get an syntax error when running ```from check-engine-lib.dqengine.validate_df import *``` due to the ```-```, is that normal? – Luiz Viola Jul 22 '22 at 08:12
You probably cant have a hyphen in module names, you will have to rename it with underscores or do any of the other points I mentioned – Tom McLean Jul 22 '22 at 08:49
I followed your sugestion but still got the same error – Luiz Viola Jul 25 '22 at 13:40

score 0 · Answer 3 · answered Jul 26 '22 at 01:37

0

As the comment said, when the method you need to import is in the same directory as the current file, you only need to import it directly.

from validate_df import _Constraint

answered Jul 26 '22 at 01:37

MingJie-MSFT

5,569
1
2
13

VS Code throws ModuleNotFoundError despite folder available

3 Answers3