3

I find myself in a situation trying to translate a TypeScript codebase to Rust, while maintaining compatibility and not breaking things.

On the TypeScript side of things, ts-proto has been used to generate bindings - and on the Rust side I've been using prost and tonic

Recently, I realized that one specific field in one of our messages has inconsistencies in it's serialization format between the two implementations. Specifically, numeric arrays are treated differently when encoding to bytes.

In ts-proto, given a numeric array field on a message, if the array is empty - it's bytes serialization includes two zero bytes

In prost, given an empty numeric array, the field is skipped entirely and not part of the serialization at all

Unfortunately I cannot change the message structure.

What I'd like to ideally do is have a way to override the encode_raw function generated through prost_build for that message type - but that message is further present as a property in other messages and isn't a top-level object by itself.

Being new to Rust I also recently learnt it's not possible to override methods in trait implementations - and ideally I don't want to fork prost and maintain that myself

Now, I'm seeking help from the community here who have more experience with both Rust and prost than I do - here's the options I find myself with right now, but please suggest more if you have any:

  • Embed a minimal typescript function that uses ts-proto for embeddings inside Rust and call it using the JS Sandbox
  • Fork prost and maintain a slightly modified version which includes empty numeric arrays in the bytes serialization in the same way as ts-proto does

What would you suggest?

Haardik
  • 185
  • 2
  • 6
  • An interesting question. As is implied in your question, this situation should not arise but it's unclear from reading it, which of the implementations is incorrect (although the Prost|Tonic implementation sounds correct: empty|default values should not be encoded). I'm mostly unfamiliar with TypeScript, please consider enriching your question with an example Message, the TypeScript code and resulting message (binary-encoded). Have you tried using another Protobuf implementation (e.g. Python, Go) to validate the behavior? – DazWilkin Aug 24 '23 at 15:14
  • 1
    Hey @DazWilkin - I dug deeper and figured out the root cause. In TypeScript, passing `null` or `undefined` to those options leads to it having similar behaviour as prost in Rust - i.e. not serializing it at all. But passing it simply an empty array instead of null or undefined causes it to still serialize it a bit. In Prost, passing an empty array will still skip serialization. Prost is "correct" here when compared to implementations in Go/Python as well. I do not have the privelege of changing the TS codebase due to compatibility reasons - so for now I forked and patched prost to match. – Haardik Aug 28 '23 at 13:37

0 Answers0