Deserialisation to Protobuf object in Python

Question

I have passed a Protobuf object in a Kafka producer and am receiving a byte array on the consumer side. Now I want to deserialize that response again back to a Protobuf object, but I am unable to do that. How can I do that?

Here is my consumer:

from email import message
from kafka import KafkaConsumer
import json
from simple import simple_message
import check_pb2

consumer = KafkaConsumer('latest', api_version=(0, 10, 1),
                         group_id='my-group', enable_auto_commit=False,
                         bootstrap_servers=['localhost:9092'])#,
           #value_deserializer=lambda m: json.loads(m.decode('ascii')))
for message in consumer:
    print(message)

I tried to use Confluent but getting

Kafka Exception : {} KafkaError{code=_VALUE_DESERIALIZATION,val=-159,str="Unknown magic byte. This message was not produced with a Confluent Schema Registry serializer"

OneCricketeer · Answer 1 · 2022-10-31T21:50:04.310

1

If you're not using Confluent Schema Registry, use ParseFromString.

for message in consumer:
    value = ParseFromString(message.value)  # example 
    print(value)

If (and only if) you are using the Schema Registry, then Confluent has a Protobuf deserializer class. See the example, which happens to also use ParseFromString internally.

edited Oct 31 '22 at 21:50

answered Jul 11 '22 at 11:55

OneCricketeer

179,855
19
132
245

tried to use confulent but getting Kafka Exception : {} KafkaError{code=_VALUE_DESERIALIZATION,val=-159,str="Unknown magic byte. This message was not produced with a Confluent Schema Registry serializer" – Ashu mishra Jul 11 '22 at 15:50
Your producer would need to use the registry as well. – OneCricketeer Jul 11 '22 at 17:31
my producer is a java service and i do not have control over it – Ashu mishra Jul 12 '22 at 05:31
can you tell me some way to do that – Ashu mishra Jul 12 '22 at 05:31
Do what? That's not required. I was explaining that you cannot use the Python confluent deserializer unless your producer used the Registry... Did you try the code I've already provided? – OneCricketeer Jul 12 '22 at 18:37
1

my java part has no schema regitry, can you tell me how can i do that without confluent deserializer ,/ – Ashu mishra Jul 13 '22 at 04:22
is there any alternative in python – Ashu mishra Jul 13 '22 at 04:22
Again, I already gave the answer how to deserialize in Python, following the Protobuf documentation saying to use the `ParseFromString` function. That's the only way I know of; there is no alternative. – OneCricketeer Jul 13 '22 at 15:12

Deserialisation to Protobuf object in Python

1 Answers1