Deserialize Kafka Thrift message with PySpark Streaming to JSON

Asked Mar 04 '22 at 08:10

Active Mar 06 '22 at 10:59

Viewed 291 times

I am using databricks environment and reading a Kafka input.

The messages consumed from Kafka are in Thrift Binary and am having issues deserialize it to JSON.

I am writing an UDF to do this, but am unable to figure out how to convert thrift -> JSON?

I have tried

import thriftpy2.protocol.json as proto
    
def decoder(thrift_data):
  return proto.struct_to_json(thrift_data)

which throws an error PythonException: 'AttributeError: 'bytearray' object has no attribute 'thrift_spec''

and also tried

def decoder(thrift_data):
  return serialize(thrift_data, protocol_factory=TSimpleJSONProtocolFactory())

which throws an error PythonException: 'AttributeError: 'bytearray' object has no attribute 'write'',

edited Mar 06 '22 at 04:11

asked Mar 04 '22 at 08:10

John Constantine

1,038
4
15
43

https://stackoverflow.com/questions/58016810/how-do-you-convert-a-thrift-object-to-json-in-python ? – Alex Ott Mar 04 '22 at 08:26
@AlexOtt Yes, I tried the methods mentioned there but they throw an error – John Constantine Mar 04 '22 at 08:41
1

Add to your post the code that you tried & error. Right now it's too generic – Alex Ott Mar 04 '22 at 08:50
Did you try this package? https://github.com/wayhome/thrift_json >>> from thrift_json import thrift2json, thrift2dict >>> json_str = thrfit2json(thrift_obj) – Benny Elgazar Mar 08 '22 at 09:58
@BennyElgazar The data is in Thrift Binary – John Constantine Mar 08 '22 at 11:03

Deserialize Kafka Thrift message with PySpark Streaming to JSON

0 Answers0