0

This is my project structure:

- config
- data
- src
    - resources 
    - db
- test

N.B.: I am using Python 3.9 and every folder that contains a .py file also has a __init__.py file

All the scripts I want to run are located in the /src folder and they used code from other scripts placed in the /src/resources folder (which is basically acting like a library). Some of these scripts also read YAML files from the /config folder

Here is the problem, I cannot find a way to properly run these scripts from the command line, I am always getting errors like:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/pi/crypto/src/ethMessage.py", line 4, in <module>
    import update_db
  File "/home/pi/crypto/src/update_db.py", line 1, in <module>
    from db.mysql_main import insertValueAndFee
  File "/home/pi/crypto/src/db/mysql_main.py", line 6, in <module>
    from src.resources.parser import read_yaml
ModuleNotFoundError: No module named 'src'

I tried both with relative and absolute import, right now absolute import is what I am using (e.g. from src.resources.parser import read_yaml)

Which is the proper way to run scripts from the command line?

EDIT:

As you suggested, I added

sys.path.append( os.path.abspath(os.path.dirname(__file__)+'/..') )

to all the main scripts, and I am still getting a similar error:

Traceback (most recent call last):
  File "src/ethMessage.py", line 6, in <module>
    import update_db
  File "/home/pi/crypto/src/update_db.py", line 1, in <module>
    from db.mysql_main import insertValueAndFee
  File "/home/pi/crypto/src/db/mysql_main.py", line 6, in <module>
    from src.resources.parser import read_yaml
ModuleNotFoundError: No module named 'src'

To clarify, I am running my script from the global folder, which in my case is named "crypto". I am also open to change the project structure with one that doesn't create problems.

Alberto Pirillo
  • 159
  • 9
  • Depends on how you are running your scripts. If you're running from a terminal, you should remove `src` from the paths. If you're running through an IDE, it depends on which one. – D. LaRocque Jul 18 '21 at 19:06
  • I am developing them on an IDE (PyCharm) that makes everything work by adding content roots to PYTHONPATH. But this solution is not portable, and does not work when I try to run the scripts with cron on Raspberry Pi. – Alberto Pirillo Jul 18 '21 at 19:44
  • And how do you open your code ? Do you open only the src folder ? Or the global folder ? – D. LaRocque Jul 18 '21 at 19:58
  • It doesn't work either way. Which one should be correct? – Alberto Pirillo Jul 18 '21 at 20:03
  • global should be. Worst case, remove src and try it : `from resources.parser import read_yml` – D. LaRocque Jul 19 '21 at 00:37

2 Answers2

0

If you want to refer to all of those packages by their root name, then all you have to do is add that folder to the Python path. So, for main program scripts in src, just add something like this:

import os
import sys
sys.path.append( os.path.abspath(os.path.dirname(__file__)+'/..') )

Now, the parent directory of your script will be on the path, no matter where you run it from. Now you can say

from src.resources.parser import read_yaml
Tim Roberts
  • 48,973
  • 4
  • 21
  • 30
  • I added it to the script and all the files it imports, and I am getting a similar error `Traceback (most recent call last): File "ethMessage.py", line 6, in import update_db File "/home/pi/crypto/src/update_db.py", line 1, in from db.mysql_main import insertValueAndFee File "/home/pi/crypto/src/db/mysql_main.py", line 6, in from src.resources.parser import read_yaml` Plus, does calling the script from a directory or another change anything? – Alberto Pirillo Jul 18 '21 at 19:47
  • Please add your new traceback to the question. We can't see the line break when you post in comments. – Tim Roberts Jul 19 '21 at 01:46
  • Sorry about that, I just edited the question – Alberto Pirillo Jul 19 '21 at 08:10
  • Did you add the "sys.path.append" line BEFORE all of your imports? – Tim Roberts Jul 19 '21 at 15:53
  • Nope, I didn't know that. I will try. Anyway, isn't there another solution? Having to paste that in every single script doesn't look very "stylish" – Alberto Pirillo Jul 19 '21 at 16:46
  • Sure. The alternative is to change your structure so the command-line scripts are all at the top level, and your auxiliary modules are all below that. That's the target model for Python. – Tim Roberts Jul 19 '21 at 17:18
  • Could you provide a sample structure of that? One that also takes into consideration the possibility to run tests that uses the same auxiliary modules as the command-line scripts. – Alberto Pirillo Jul 19 '21 at 17:35
0

If someone is still looking for a solution, I highly recommend not to bother with Python's imports: they are probably the worst part of the whole language.

Instead, if you want to use some files as a library, you should use setuptools to create a package from those files.

Then, you can install it locally or publish it on PyPi.

This way, you can import your library in a script just like another third-party module, (e.g. requests, selenium, ...), and things will work, instead of giving you a headache because a file is in a directory instead of another.

Alberto Pirillo
  • 159
  • 9