2

So I have to create a run folder, that is, a folder which has a bunch of python files which I need to run. I am able to create this folder with ease and all the files are there. However, when I try to run the files using importlib, python is not about to find it.

I want to make sure that the code running is, in fact, the run folder code, hence I change the directory to location of the run folder. 

abs_path = os.path.abspath("{}".format(run_location))
os.chdir(abs_path)
files = [f for f in os.listdir('.') if os.path.isfile(f)]
print(files)
try:
    driver_module = importlib.import_module("main_driver.py")
    driver_module.main(config, logger)
except Exception as e:
    logger.error("error", str(e))
    finish_fail(config, logger)
finish_success(config, logger)

In the above example, I just want to run main_driver.py. The output from the above is:

['PrepareDataframe.py', 'categorical_encoding.py', 'extra_files.zip', 'build_features.py', 'spot_extractor.py', 'dev.ini', 'featuriser.py', 'main_driver.py', 'time_features_extract.py']

Clearly, main_driver.py is within the current working directory but I get this error. 

No module named 'main_driver'

Traceback: 

Traceback (most recent call last):
  File "./utils/submit.py", line 292, in <module>
    driver_module = importlib.import_module("main_driver")
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'main_driver'
Bryce Ramgovind
  • 3,127
  • 10
  • 41
  • 72
  • 1
    Don't use the `.py` extension on the filename. Just use `import_module('main_driver')`. – John Gordon Sep 10 '19 at 12:18
  • Still gives the same issue `driver_module = importlib.import_module("main_driver")` – Bryce Ramgovind Sep 10 '19 at 12:22
  • That's odd; it works for me. Can you try a smaller example script that leaves out all the chdir stuff and just has `importlib.import_module('main_driver')`, and place it in the same directory as that module? – John Gordon Sep 10 '19 at 12:29
  • @BryceRamgovind Try importing it without the `.py` extension. `driver_module = importlib.import_module("build_features")` – lahsuk Sep 10 '19 at 12:44
  • @lahsuk still gives that import error even without the '.py' extension. The error disappears when I run it directly from the run folder and not chdir. – Bryce Ramgovind Sep 10 '19 at 12:47
  • @JohnGordon the error disappears when I run it from the run folder directly without chdir – Bryce Ramgovind Sep 10 '19 at 12:47

1 Answers1

8

This appears to be a PYTHONPATH issue.

When you run the Python interactive interpreter, the first item in sys.path is an empty string, which effectively means "the current directory". So any time you try to import a module, it will look in the current directory first.

However, if you run python myscript.py, this is not the case -- the first item in sys.path is the actual directory you were in when you ran the script. So, if you use os.chdir() to change the current directory, the new current directory isn't in sys.path, and so import_module('foo') fails because it's not found.

If you want to mimic the interactive python behavior of always looking in the current directory for modules, add sys.path.insert(0, '') to your script.

John Gordon
  • 29,573
  • 7
  • 33
  • 58
  • worked like a charm, thank you. are there any better ways to approach this? for my research I'm often building a self contained module where I sometimes run files inside folders of the module so I have to move the cwd up a folder or two, it's not the cleanest of code but it works, I'm just wondering if there are better or cleaner approaches. – Flying Turtle Aug 14 '23 at 07:05