1

Messages are failing to deserialise from bytes to any.Any with the error message: proto: cannot parse invalid wire-format data but only when all the message fields have values, it's as if there's a size limit on the messages, empty or partially empty messages deserialise fine, but if all fields have values it will fail. The messages are coming from a Dapr adapter to a redis stream.

import (
"github.com/golang/protobuf/proto"
"github.com/golang/protobuf/ptypes/any"
)

func (h *EventHandler) instanceHandler(ctx context.Context, e *common.TopicEvent) (bool, error) {

    var anyMsg any.Any
    b := []byte(e.Data.(string))
    err := proto.Unmarshal(b, &anyMsg)
    if err != nil {
        h.logger.Error().
            Str("eventId", e.ID).
            Str("topic", e.Topic).
            Str("source", e.Source).
            Err(err).
            Msg("")

        return false, err
    }

The above will fail if all the message fields have values, message example:

message GetElementsPaginatedRequest {
  string eid = 1;
  string mediatorID = 2;
  string callerID = 3;
  string accountID = 4;
  string streamID = 5;
  int32 limit = 6;
  int32 page = 7;
  string order = 8;
  string clientID = 9;

}

Serialise before sending over the wire:

    err := anyMsg.MarshalFrom(&epb)
    if err != nil {
        return err
    }

    b, err := proto.Marshal(&anyMsg)
    if err != nil {
        return err
    }

If I remove one field (any field) for example, GetElementsPaginatedRequest.accountID, deserialisation will succeed.

Testing serialisation and deserialisation also works fine within a local test, this behaviour just seems to occur for messages received from the Dapr adapter.

Unable to replicate this issue for messages that are not received via Dapr and redis. In a local test deserialisation works fine, regardless of the size of the message.

sevvaio
  • 21
  • 3
  • Please include a hex dump of the data that fails to decode. And because the error seems to occur in transit, compare if the data you get from protobuf encoder matches the data at the other end you are trying to decode. – jpa Mar 11 '23 at 19:01
  • ok just compared, the sender encodes to 4415 bytes, but the receiver is trying to decode 6196 bytes. After removing some values and sending 292 bytes, the receiver also gets 292 bytes and it decodes. – sevvaio Mar 11 '23 at 19:33

1 Answers1

1

This bytes size difference behaviour can be avoided by setting the content-type on the publisher client and changing the receiver to unmarshal from TopicEvent.Data to TopicEvent.RawData (which is possible after setting the context-type).

New publisher configuration

opts := []dapr.PublishEventOption{
        dapr.PublishEventWithContentType("application/octet-stream"),
    }
    err = client.PublishEvent(ctx, pubsubName, topic, b, opts...)

And proto.Unmarshal from the RawData field.

err := proto.Unmarshal(e.RawData, &anyMsg)

The bytes match and the receiver can decode.

sevvaio
  • 21
  • 3