Creating Python protocol buffer package using gradle plugin

Question

We are already using Gradle to generate the code for Kotlin protocol buffers, and we use Google's protobuf gradle plugin to do this. On the data / analytics side, we need to work with Python, but there is a nice gradle tool generate the language-specific files for both Kotlin and Python (and others).

To make it happen, there is a simple gradle build addition (in Kotlin):

protobuf {
    protoc {
        artifact = "com.google.protobuf:protoc:VERSION"
    }
    plugins{
        id("grpc") {
            artifact = "io.grpc:protoc-gen-grpc-java:VERSION"
        }
        id("grpckt") {
            artifact = "io.grpc:protoc-gen-grpc-kotlin:VERSION:jdk8@jar"
        }
    }
    generateProtoTasks {
        all().forEach { task ->
            task.plugins {
                id("grpc")
                id("grpckt")
            }
            task.builtins {
                id("kotlin")
                id("python") ### This is all that's needed to build it in Python, too!!
            }
        }
    }
}

As I commented above, to have this also generate the protobufs in Python, I simply add id("python") and it is built.

However, the generated ..._pb2.py files are not structured how I need, in order to be able to package them up.

If the .proto files folder structure is like this:

> tree
├── order
│   ├── pricesummary
│   │   └── v1
│   │       └── price_summary.proto
│   ├── service
│   │   └── v1
│   │       └── order_service.proto
│   └── v1
│       └── order.proto

then the generated file structure is this:

.
├── order
│   ├── pricesummary
│   │   └── v1
│   │       └── price_summary_pb2.py
│   ├── service
│   │   └── v1
│   │       └── order_service_pb2.py
│   └── v1
│       └── order_pb2.py

And if I look into one of the files that references another file, I will see something like this:

from order.pricesummary.v1 import price_summary_pb2 as order_dot_pricesummary_dot_v1_dot_price__summary__pb2

But now I need to make it into a package. So let's say the whole package becomes something like:

.
├── pdm.lock
├── pyproject.toml
├── README.md
├── src
│   └── myprotobuf
│       ├── __init__.py
│       ├── order
│       │   ├── pricesummary
│       │   │   └── v1
│       │   │       ├── price_summary_pb2.py
│       │   ├── service
│       │   │   └── v1
│       │   │       └── order_service_pb2.py
│       │   └── v1
│       │       ├── order_pb2.py

When I perform pdm build (or poetry build or whatever flavor you prefer), the package will now become myprotobuf. I will need to import it as myprotobuf. But most importantly, internal to the package, it will also need to reference myprotobuf. So, to revisit the line of Python above, instead of saying

from order.pricesummary.v1 import price_summary_pb2 as order_dot_pricesummary_dot_v1_dot_price__summary__pb2

I need for the Python code generated by protoc to say

from myprotobuf.order.pricesummary.v1 import price_summary_pb2 as order_dot_pricesummary_dot_v1_dot_price__summary__pb2

Notice the from myprotobuf.order... instead of from order...

---

I want to be able to ensure that all of these imports that are internal to the package use the top package name, so that they can be properly resolved upon install on other devices.

Is there any way to:

1. Force protoc to prepend the imports so that the generated files can be used as a package?
   • I saw some issues submitted by other folks having very similar problems, but nothing quite the same:
     • https://github.com/protocolbuffers/protobuf/issues/7061
     • https://github.com/protocolbuffers/protobuf/issues/9667
2. Find another way that I can build a package properly? The problem is that, while I only show order at the top level, as if there is only one top level, there are in fact several folders at that top level.
   • Maybe the best solution is just to create another level one higher up and call it myprotobuf so that the proper structure is captured. I still would have to adjust the package structure to remove the extra myprotobuf folder that will be created.
3. Any better ideas?