1

I want to generate _pb2.py files from .proto files and use them for bigquery data upload. The proto file contains simple messages with only string and int32 fields like:

syntax = "proto2";

package duplicate;

message duplicate {
    required int32  id = 1;
    optional string title = 2;
...

I need to use a new protoc installation. Thus I followed these installation instructions.

**Some possibly relevant details about the installation: **

$ python -V
Python 3.10.7

$ bazel --version
bazel 5.3.2-homebrew

protoc --version
libprotoc 3.21.9

from protoc-21.9-osx-universal_binary (MacOS Monterey, version 12.5.1)

  • tests ran fine (Output: Ran 911 tests in 7.674s, OK (skipped=9))
  • brew unlinking was executed successfully
  • installation via setup.py did not show any problems

when I now run protoc, I get a _pb2.py file which contains something like:

from google.protobuf.internal import builder as _builder
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import symbol_database as _symbol_database
# @@protoc_insertion_point(imports)

_sym_db = _symbol_database.Default()

DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x19duplicate.proto\x12\x13duplicate\"\xb...........

_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'duplicate_pb2', globals())
if _descriptor._USE_C_DESCRIPTORS == False:

  DESCRIPTOR._options = None
  _DUPLICATE._serialized_start=51
  _DUPLICATE._serialized_end=498
# @@protoc_insertion_point(module_scope)

Problem:

  • Comparing to the _pb2.py generated by an older version (which I cannot use anymore), there are some parts missing, precisely:
_DUPLICATE = ....
duplicate = _reflection.GeneratedProtocolMessageType( .......

=> bigquery upload fails.

Questions:

  1. Since running protoc did not throw any errors, I first want to make sure that the assumption is right, that the resulting _pb2.py file is incomplete. It also looks like that because _DUPLICATE is used but never assigned or imported.

  2. Does anyone have an idea of where it is going wrong and how I could solve it?

Please let me know, if you need more details about my procedure. Thanks a lot!

ChrisChros
  • 11
  • 2
  • Did you try it with the grpc_tools compiler? `pip install grpcio-tools`, then `python -m grpc_tools.protoc -I/path/to/folder --python_out=. --pyi_out=. --grpc_python_out=. whatever.proto` – stanvooz Dec 07 '22 at 20:09

1 Answers1

0

I've got the same problem. In my case, generating the pb2 files makes my old code fail as the generated files are not exporting properly the classes, they have to be directly accessible through the pb2 class:

Example new import

In my case, I've got protoc:3.21.8. As a next step, I am trying to find out why the new pb2 does not export the classes. I hope it helps.

Willie Cheng
  • 7,679
  • 13
  • 55
  • 68
Jose
  • 1
  • Hi Jose, thank you for your answer. I found that there was actually no problem with the _pb2.py files, but with something else that I had overlooked regarding the bigquery upload. I was also mislead by my development environment showing me unresolved references when using the pb2s, but in fact the global variables do the job as far as I see it now. All the best! – ChrisChros Oct 28 '22 at 15:10