3

I can create the schema (Descriptors.Descriptor) at runtime dynamically using FileDescriptorProto, also I'm able to serialize and deserialize messages using DynamicMessage.

However performance of DynamicMessage is not good enough because of the way it constructs the messages. I wonder whether it's possible to compile the schema at runtime and use it when de-serializing messages for better performance.

If protocol buffers doesn't provide a way to compile the schemas at runtime then if it's possible to convert the Descriptors.Descriptor to a temporary .proto file then I can try to generate the classes by calling protoc command from my program and load them to the program using Class.forName API.

Boyolame
  • 329
  • 2
  • 13
  • can You pls share an eg. or working code where You were able to create schema dynamically at runtime using FileDescriptor. I am facing a similar situation to yours. We have a string that represents the whole message class(normally contained in .proto file) and I want to generate a class that I can use for reading messages serialized using the above message class. – Vipresh Oct 08 '20 at 17:25
  • I am facing a similar problem , curious what solution you ended up with ? – user179156 Mar 15 '21 at 20:33

1 Answers1

5

Protocol Buffers does not include any kind of built-in "JIT" compiling.

If you want to generate code and compile it with javac at runtime, and you're starting with a FileDescriptorProto, then probably what you'd like to do is invoke the Java code generator directly (without converting the proto definition into text).

You could accomplish this by writing a little C++ program that creates a google::protobuf::compiler::java::JavaGenerator and passes it into google::protobuf::compiler::PluginMain(). You now have a program which reads a CodeGeneratorRequest from standard input and writes a CodeGeneratorResponse to standard output. So, you can execute this program to generate Java code directly from FileDescirptorProtos. It's up to you to do pass this code on to the Java compiler. This should be much easier and cleaner than trying to produce text to pass to protoc.

Kenton Varda
  • 41,353
  • 8
  • 121
  • 105
  • Unfortunately I don't have enough experience with C++ so I think it would be hard for me to maintain and distribute that plugin into the nodes in the cluster. javac compiler doesn't actually compile the code on the fly so even if I read the source code directly from standard input, I will still need to write it into a file in order to use javac. I thought maybe protobuf can provide a way to serialize FileDescriptorProto back to .proto definition since it already uses that definition in order to construct FileDescriptorProto, but if it's not then I will go with your way. – Boyolame Feb 08 '15 at 11:12
  • FWIW, the necessary C++ code here is about four lines long for the whole program -- it's just creating a `JavaGenerator` and passing it to `PluginMain`. ;; I'm somewhat confused by your objection to `javac` -- even if you converted the FileDescriptorProto back to text, and then fed it into `protoc`, the output is still Java code, so you still need `javac` or something like it. – Kenton Varda Feb 10 '15 at 04:32
  • I assumed the reason that you suggested me to write a small plugin is the portability issue because the program would allow me to write/read data to the program directly without using filesystem. However even though I read the java source code from that program, I still need to write it to filesystem in order to run javac compiler so the only benefit of that plugin for me is converting FileDescriptorProto instance to protoc definition is not required anymore. Also protoc is easy to install via package managers in linux but if I write a plugin I also have to distribute it to the nodes myself. – Boyolame Feb 11 '15 at 13:46
  • No, the reason I suggest the plugin approach is because it would be much easier and cleaner than trying to convert `FileDescriptorProto` back into a `.proto` file. :) I suspect you will have a lot more trouble (write a lot more code, have a lot more bugs) with that approach. – Kenton Varda Feb 12 '15 at 07:57
  • Oh, I got it now. Thanks for the suggestions, I will give it a try. :) – Boyolame Feb 12 '15 at 12:57
  • did it work ? would like to understand your final solution – user179156 Mar 15 '21 at 20:33