Protoc.py compiles files differently depending on where it is called from

Question

If protoc.py is called from powershell, the compiled file is missing about half of the code as opposed to when protoc.py is called from natively from python. I have a powershell script and a python script. My aim is to remove the python script so I only need the powershell script.

My python script is here:

PythonScript.py

import os
from grpc_tools.protoc import main

os.chdir("<MyProtoDirectory>")

main([
    '--proto_path=.',
    '--python_out=<MyOutputDirectory>',
    '--grpc_python_out=<MyOutputDirectory>',
    '<MyProto'])

and my powershell script is here:

PowershellScript.ps1

$tmp = (Get-Location).tostring()
Set-Location -Path ('<MyProtoDirectory>')

try {
  python '...\protoc.py' `
    --proto_path='.' `
    --python_out='<MyOutputDirectory>' `
    --grpc_python_out="<MyOutputDirectory>" `
    "<MyProto>"

}
finally {
  Set-Location -Path $tmp
}

I am calling main in python, and the script in powershell, so I modified protoc.py to be:

protoc.py

if __name__ == '__main__':
    #proto_include = pkg_resources.resource_filename('grpc_tools', '_proto')
    #sys.exit(main(sys.argv + ['-I{}'.format(proto_include)]))
    a = main(sys.argv[1:])
    print(a)
    sys.exit()

where I have commented out the altered lines. When I run the python script, I get the result:

outcome_pb.py        67kb
outcome_pb_grpc.py   77kb

When I run the powershell script, I get the result:

outcome_pb.py        28kb
outcome_pb_grpc.py   77kb

The powershell output outcome_pb.py is incorrect and cannot be run, whereas the python outcome is good. It is worth noting outcome_pb_grpc is identical when run using either the powershell script or the python script. Futhermore, I can write a new powershell script that simply calls the python script:

python .\PythonScript.py

and this leads to the same issue. I need both outcome_pb.py and outcome_pb_grpc.py for my application. Why is this happening?

I suspect (!) that you're catching different versions of `protoc` (and thus its internal plugin for python that effects `python_out`). Google has changed how protobuf code is generated for Python quite considerably and this would explain the file size differences and why you can't compile stub code generated by a different `protoc` compiler version. gRPC has not changed (as significantly). For example, see [Changes made 06-May-2022](https://developers.google.com/protocol-buffers/docs/news/2022-05-06). But there was also a significant change (somewhere) between e.g. 3.14.0 and 3.20.3. — DazWilkin, Nov 11 '22 at 17:40

score 0 · Accepted Answer · answered Nov 12 '22 at 08:31

0

Like DazWilkin said in a comment, it is likely that there is a difference in either the grpcio_tools library or the Python version you end up using.

To test, you can use some prints:

 import sys
 import grpc_tools.protoc
 print("Python version: %s" % sys.version)
 print("grpcio_tools path: %s" % grpc_tools.protoc.__file__)

and compare the output between the working and not-working case.

answered Nov 12 '22 at 08:31

jpa

10,351
1
28
45

Thank you, the version of grpcio_tools that seemed to fail was 1.50.0, rolling back to 1.48.1 allowed me to compile the files correctly. – George Gayton Nov 14 '22 at 14:33

Protoc.py compiles files differently depending on where it is called from

1 Answers1