How to generate python class files from protobuf

Question

I am trying to transfer large amounts of structured data from Java to Python. That includes many objects that are related to each other in some form or another. When I receive them in my Python code, it's quiet ugly to work with the types that are provided by protobuf. My VIM IDE crashed when trying to use autocomplete on the types, PyCharm doesn't complete anything and generally it just seems absurd that they don't provide some clean class definition for the different types.

Is there a way to get IDE support while working with protobuf messages in python? I'm looking at 20+ methods handling complex messages and without IDE support I might as well code with notepad.

I understand that protobuf is using metaclasses (although I don't know why they do that). Maybe there is a way to generate python class files from that data or maybe there is something similar to typescript typing files.

Did I maybe misuse protobuf? I believed I would describe my domain model in a way that may be used across languages. In Java I am happy with the generated classes and I can use them easily. Should I maybe have used something like swagger.io instead?

score 17 · Answer 1 · answered Oct 31 '19 at 04:38

If you are using a recent Python (3.7+) then https://github.com/danielgtaylor/python-betterproto (disclaimer: I'm the author) will generate very clean Python dataclasses as output which will give you proper typing and IDE completion support.

For example, this input:

syntax = "proto3";

package hello;

// Greeting represents a message you can tell a user.
message Greeting {
  string message = 1;
}

Would generate the following output:

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# sources: hello.proto
# plugin: python-betterproto
from dataclasses import dataclass

import betterproto


@dataclass
class Hello(betterproto.Message):
    """Greeting represents a message you can tell a user."""

    message: str = betterproto.string_field(1)

In general the output of this plugin mimics the *.proto input and is very easy to read if you happen to jump to definition on a message or field. It's been a huge improvement for me personally over the official Google compiler plugin, and supports async gRPC out of the box as well.

Why does `betterproto` uses custom fields for data classes instead of python's built-in [`field`](https://docs.python.org/3/library/dataclasses.html#dataclasses.field) ? — moshevi, Jun 02 '20 at 12:18
@moshevi internally it is actually using `dataclasses.field`. The field functions are just shortcuts. See https://github.com/danielgtaylor/python-betterproto/blob/eec24e4ee85c581d5185c258d2ea2cbb4d39be20/betterproto/__init__.py#L175-L180 — Daniel, Jun 04 '20 at 21:53
but this way you are trying my code to protobuff. It would make sense to have a data access layer Wich outs technology agnostic python types and dataclasses — moshevi, Jun 12 '20 at 23:54

score 3 · Answer 2 · answered Apr 10 '18 at 17:54

3

As of now, nothing like that is available. You might want to follow this issue: https://github.com/google/protobuf/issues/2638 to be up to date.

answered Apr 10 '18 at 17:54

Michał

2,202
2
17
33

1

While I didn’t find a solution, I did find this which generates type hint files https://github.com/dropbox/mypy-protobuf – pascalwhoop Apr 11 '18 at 14:54
@pascalwhoop - Does `mypy-protobuf` works with protoc 2.5.0 and python 2.7.6 ? – tuk Sep 18 '18 at 12:06
@tuk not sure about the exact versions but it did work for my project. – pascalwhoop Sep 19 '18 at 13:03
1

The issue is already resolved and the new options is `--pyi_out=`. Perhaps you could update your answer. I find it far more better than the others. – Zaffy Jun 01 '22 at 15:35

score 2 · Accepted Answer · answered Sep 23 '18 at 07:22

2

mypy-protobuf generates the type hint files. But as discussed here this works only from protobuf 3.0 and python 2.7 onwards.

answered Sep 23 '18 at 07:22

tuk

5,941
14
79
162

score 0 · Answer 4 · edited Mar 19 '23 at 09:51

What "-Zaffy" said worked for me. (--pyi_out)

EXAMPLE

Note : Protobuf filename example = addressbook.proto

CLI Cmd to create intellsense info (compiles in this same dir)

protoc -I . --pyi_out=. addressbook.proto

Then create python code. (compiles in this same dir)

protoc -I . --python_out=. addressbook.proto

This will create a addressbook_pb2.py and addressbook_pb2.pyi

then VS code away!

How to generate python class files from protobuf

4 Answers4

What "-Zaffy" said worked for me. (--pyi_out)

EXAMPLE