GRPC_TOOLS.PROTOC generates broken pb2 python file

Question

I am trying to build an application using gRPC in Python. I have the following directory layout:

service/

├─ auth/

│  ├─ auth_pb2_grpc.py

│  ├─ auth_pb2.py

│  ├─ auth.py

│  ├─ client.py

├─ flask/

├─ todo/

├─ app.py

├─ auth.proto/

├─ generate.sh/

├─ requirements.txt

I set up the code for proto file in the following way:

syntax = "proto3";
package auth;


message RegisterRequest {
    string username = 1;
    string password = 2;
}

message RegisterResponse {
    bool success = 1;
    string message = 2;
}

message LoginRequest {
    string username = 1;
    string password = 2;
}

message LoginResponse {
    bool success = 1;
    string message = 2;
    string token = 3;
}

message ValidateTokenRequest {
    string token = 1;
}

message ValidateTokenResponse {
    bool success = 1;
    string message = 2;
}

service AuthService {
    rpc Register(RegisterRequest) returns (RegisterResponse) {}
    rpc Login(LoginRequest) returns (LoginResponse) {}
    rpc ValidateToken(ValidateTokenRequest) returns (ValidateTokenResponse) {}
}

when I run

python -m grpc_tools.protoc -I. --python_out=./auth --grpc_python_out=./auth ./auth.proto

protobuf generates the file into auth subdirectory.

Seems good, ain't it? However, when I run app.py to test newly created files, it returns -

ModuleNotFoundError: No module named 'auth_pb2'

Here are the insides of 'auth_pb2' file:

    # -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler.  DO NOT EDIT!
# source: auth.proto
"""Generated protocol buffer code."""
from google.protobuf.internal import builder as _builder
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import symbol_database as _symbol_database
# @@protoc_insertion_point(imports)

_sym_db = _symbol_database.Default()




DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\nauth.proto\x12\x04\x61uth\"5\n\x0fRegisterRequest\x12\x10\n\x08username\x18\x01 \x01(\t\x12\x10\n\x08password\x18\x02 \x01(\t\"4\n\x10RegisterResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\"2\n\x0cLoginRequest\x12\x10\n\x08username\x18\x01 \x01(\t\x12\x10\n\x08password\x18\x02 \x01(\t\"@\n\rLoginResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\r\n\x05token\x18\x03 \x01(\t\"%\n\x14ValidateTokenRequest\x12\r\n\x05token\x18\x01 \x01(\t\"9\n\x15ValidateTokenResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t2\xca\x01\n\x0b\x41uthService\x12;\n\x08Register\x12\x15.auth.RegisterRequest\x1a\x16.auth.RegisterResponse\"\x00\x12\x32\n\x05Login\x12\x12.auth.LoginRequest\x1a\x13.auth.LoginResponse\"\x00\x12J\n\rValidateToken\x12\x1a.auth.ValidateTokenRequest\x1a\x1b.auth.ValidateTokenResponse\"\x00\x62\x06proto3')

_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'auth_pb2', globals())
if _descriptor._USE_C_DESCRIPTORS == False:

  DESCRIPTOR._options = None
  _REGISTERREQUEST._serialized_start=20
  _REGISTERREQUEST._serialized_end=73
  _REGISTERRESPONSE._serialized_start=75
  _REGISTERRESPONSE._serialized_end=127
  _LOGINREQUEST._serialized_start=129
  _LOGINREQUEST._serialized_end=179
  _LOGINRESPONSE._serialized_start=181
  _LOGINRESPONSE._serialized_end=245
  _VALIDATETOKENREQUEST._serialized_start=247
  _VALIDATETOKENREQUEST._serialized_end=284
  _VALIDATETOKENRESPONSE._serialized_start=286
  _VALIDATETOKENRESPONSE._serialized_end=343
  _AUTHSERVICE._serialized_start=346
  _AUTHSERVICE._serialized_end=548
# @@protoc_insertion_point(module_scope)

As I see the problem is in protobuf generation because p2b file is somehow not complete. No clue why it occurs...

Also, there was a similar question grpc_tools.protoc generates python files with broken imports, but there was no explicit answer.

Could you please hint on where I might be wrong in my code?

score 2 · Answer 1 · answered Oct 17 '22 at 07:43

I have had the same issue when generating the p2b file. I have been able to solve it by degrading the next packages:

Protobuf==3.14.0
grpcio-tools==1.33.2

I think that there must be a bug on the new releases of grpcio-tools and protobuf. I started using grpc in python on december 2020 and the versions I used on that time works fine.

score 1 · Answer 2 · answered Nov 06 '22 at 13:25

1

Found one more stable version, which works with python:3.9-alpine

grpcio-tools==1.46.5
Protobuf==3.19.5

answered Nov 06 '22 at 13:25

Elena Kutanov

11
1

score 0 · Answer 3 · answered Oct 03 '22 at 18:55

I think you're encountering this issue with Python code-generation.

You don't include the proto file but I suspect (?) you're not generating the services/messages into a package {foo};.

In this case, I recommend that you put the generated sources into the root directory (not ./auth). import'ing the file should then work.

python3 -m grpc_tools.protoc \
--proto_path=${PWD} \
--python_out=${PWD} \
--grpc_python_out=${PWD} \
${PWD}/auth.proto

If you don't want this structure, you will need to:

edit package names
import's; and
create package __init__.py's

Please don't include screenshots on Stack overflow. Screenshots may not outlive the Stack overflow content and they inhibit copy-and-paste. You can replace screenshots by copying-and-pasting text in the first place.

Thank you for your answer! I've updated the question according to the standard and added some information for clarification. Now you can see the proto file and I am doing it correctly, however, it does not help the cause. Still. the issue does not go away. Perhaps, you have any other theories on why it may happen? — Qagan, Oct 04 '22 at 08:44

GRPC_TOOLS.PROTOC generates broken pb2 python file

3 Answers3