0

Currently, I have a use case where I need to take messages from RabbitMQ Message Bus, append the message size (in bytes) and output message using HDFS Sink.

To start, I have created my own processor which appends the size to the message. The reason I do this is because the encoding needs to be that of a Google Protocol Buffer.

My app looks like the following:

stream create --name rabbit-to-hdfs --definition "rabbit | delim-protobuf | hdfs "

When HDFS Sink outputs the message I see [B@12768762. I have Google'd around and have seen recommendations to add the following: 

spring.cloud.stream.bindings.input.consumer.headerMode=raw

However, this does not see to help me at all ! That said, if I change the app to go to a file using the following:

[input | processor ] | file --binary=true

Then, everything works fine. However, I like the rollover features offered by the HDFS Sink.

Any ideas ?

Dennis Jansky
  • 107
  • 12

2 Answers2

0

File is working because it's just dumping the bytes it received, however looking at the HDFS sink, it seems it needs to consume a java.io.Serializable object as the input. But in your case you are sending a byte array from a protobuf object (I'm assuming that's what's going on)

Vinicius Carvalho
  • 3,994
  • 4
  • 23
  • 29
  • Yes, the processor that I implemented returns a byte array of the GPB. Are you saying that it should return a java.io.Serializable ? – Dennis Jansky Sep 22 '17 at 17:13
  • I tried "stream deploy --name rabbit-to-log --properties "app.log.spring.cloud.stream.bindings.input.content-type=application/x-java-object;type=java.io.Serializable"" which did not work. Are you saying that the processor should return a java.io.Serializable opposed to byte[] – Dennis Jansky Sep 22 '17 at 17:25
  • I re-wrote the processor to return an Envelope object which inherits from com.google.protobuf.GeneratedMessageV3. This class in-turn inherits from Serializable. When un-register/register the new app and begin to process data I receive an error message "Unable to deserialize [CUAVProtos$Envelope] using the contentType [application/x-java-object;type=CUAVProtos$Envelope] CUAVProtos$Envelope" – Dennis Jansky Sep 22 '17 at 19:04
0

The types are not compatible that's the problem. By setting that contentType in SCS you are just asking the framework to use java serialization to call writeObject. But since you are using protobuf which is already a serialization framework it would not work. The issue here is that the sink really seems (I'm not familiar with the sink code) to expect a Serializable, but you are not providing one. What you could do is modify the sink app or provide a custom converter that knows how to convert from protobuf to Serializable, don't even know if that makes sense to be honest.

Vinicius Carvalho
  • 3,994
  • 4
  • 23
  • 29