python3 resolve relative import with internal dependency inside module

Question

My folder structure:

grpc/
    server.py
    client.py
    ...    
    src/
        __init__.py
        image_pb2.py
        image_pb2_grpc.py 
        ...

server.py

import image_pb2_grpc
import image_pb2

...

image_pb_grpc.py

import image_pb2
...

In summary, server depends on both image_pb2 and image_pb2_grpc, while image_pb2_grpc also depends on image_pb2.

Now, if I move my server.py inside the src folder, the code surely runs fine, as everything is in the path.

The problem is server.py should be outside the src folder.

Now, I can still solve the problem by changing server.py and image_pb2_grpc.py with:

server.py

from src import image_pb2_grpc
from src import image_pb2

...

image_pb_grpc.py

from src import image_pb2
...

The issue with this approach is, I need to change image_pb_grpc.py manually, as they are generated code from grpc, and they are generated using a bash script, so it is not possible to change them manually.

How can I organize my project so that I can run server outside src, while not changing image_pb2 or image_pb2_grpc?

I am not sure, but try `from .src import ...` in `server.py` and `from . import image_pb2` in `image_pb_grpc.py`. Also add a empty `__init__.py` to the `src` folder. — user_na, Jun 24 '21 at 20:26
@user_na as I told in my question, I can not change image_pb2 file as it's auto-generated. — Zabir Al Nazi, Jun 24 '21 at 20:42

score 0 · Answer 1 · answered Jun 24 '21 at 20:26

0

I would suggest to add to your src/__init__.py following code:

import image_pb2_grpc
import image_pb2

Then your server.py will be able to import those both files simply by:

from src import image_pb2_grpc, image_pb2`

I haven't tried it but it should work.

answered Jun 24 '21 at 20:26

Maciej M

786
6
17

Thanks for the answer, I tried it before, didn't work. – Zabir Al Nazi Jun 24 '21 at 20:40
How about changing `PYTHONPATH` to the `src/`? – Maciej M Jun 24 '21 at 20:42
I see, I tried `sys.path.append` but it didn't work. But when I do both, `sys.path.append` and importing inside `__init__.py` it works. – Zabir Al Nazi Jun 24 '21 at 20:46
I mean something like that in the shell `$ export PYTHONPATH:=./src` – Maciej M Jun 24 '21 at 20:48
But that will be temporary, right? Wouldn't I need to run `export` each time I run `server.py` – Zabir Al Nazi Jun 24 '21 at 20:54
1

I will work as long your environment (here it's shell) will not get restarted. Even though, you can always add this to the default shell values and it will be always included in the `PYTHONPATH` – Maciej M Jun 24 '21 at 20:57

Mr_and_Mrs_D · Answer 2 · 2021-06-25T13:50:12.520

0

The correct way of running your script is:

$ cd grpc
$ python -m server # this is the one you want to run

Use the relative imports in server.py.

If this does not work I am afraid you need to hack sys.path - but this should be avoided at all cost

EDIT:

In server once you import the image_pb2 as in from .src import image_pb2 do

sys.modules['image_pb2'] = sys.modules['src.image_pb2']

then import image_pb2_grpc. Do not add any imports in __init__.py

edited Jun 25 '21 at 13:50

answered Jun 25 '21 at 09:23

Mr_and_Mrs_D

32,208
39
178
361

This seems like a better option, but I can't seem to get it working. If I cd into the directory, then the `__init__.py` imports fail, and I can't seem to import the libraries. – Zabir Al Nazi Jun 25 '21 at 12:53
What are the `__init__` imports? – Mr_and_Mrs_D Jun 25 '21 at 12:58
Initially, I solved by importing `image_pb2` and `image_pb2_grpc` inside `src/__init__.py`, and adding `sys.path.append(src path)` inside `server.py`. If I try to run `python3 ../server.py` after `cd`ing into `src`, I get error in the `__init__.py` and `server.py`, saying no module named ... – Zabir Al Nazi Jun 25 '21 at 13:01
You should cd into parent dir not src. – Mr_and_Mrs_D Jun 25 '21 at 13:23
Then you need some hacks in src probably – Mr_and_Mrs_D Jun 25 '21 at 13:24

python3 resolve relative import with internal dependency inside module

2 Answers2