0

I am trying to use the outbox event router of debezium for mongodb. The consumer is a spring cloud stream application. I cannot deserialize the message because spring cloud expects the message id header to be UUID, but it receives byte[]. I have tried different deserializers to no avail. I am thinking of renaming the id header in order to skip this spring cloud check, or remove it altogether. I have tried the ReplaceField SMT but it does not seem to modify the header fields.

Also is there a way to overcome this in spring?

IonutB
  • 71
  • 1
  • 6
  • ID in Message header is required and immutable. I am not sure what serializer or router you are referring to. I also don't understand how your consumer is s-c-stream application as it would require a binder and we don't have one for mongo – Oleg Zhurakousky May 12 '22 at 10:20
  • Ok let me try to make it clearer...I am using the mongodb connector for debezium. This will send messages to kafka whenever I change the data. After the messages are sent to kafka, I am attaching a @StreamListener to consumer the outbox messages. That fails because it requires the id header to be UUID. It will fail to trigger the listener if you use condition = "headers['som header']=='some value'". I do not use the api from Debezium. It is deployed as a connect docker image. I jsut connect to the kafka topic. The router is MongoDB Outbox Event Router – IonutB May 12 '22 at 10:24
  • I have used the DropHeader transformation and it seems the id header is removed. But know I get "Cannot find a @StreamListener matching for message with id: null" error – IonutB May 12 '22 at 10:27
  • 1
    You must be using a very outdated spring-cloud-stream as @StreamListener is long deprecated and is already removed from the main branch. All the documentation about it was removed several years ago. Please update to the latest version where I am sure the header as byte[] issue has been fixed - https://docs.spring.io/spring-cloud-stream/docs/current/reference/html/ – Oleg Zhurakousky May 12 '22 at 11:07

1 Answers1

1

The solution to the initial question is to use the DropHeaders SMT(https://docs.confluent.io/platform/current/connect/transforms/dropheaders.html). This will remove the id header that is populated by debezium.

But as Oleg Zhurakousky mentioned, moving to a newer version of spring-cloud-stream without @StreamListener solves the underlying problem. Apparently @StreamListener checks if a message has an id header and it demands to be of type Uuid. By using the new functional way of working with spring-cloud-stream, the id header is actually overwritten with a new generated value. This means that the value populated by debezium (the id column form the outbox table) is ignored. I guess if you need to check for duplicate delivery, maybe it is better to create your own header instead of using the id. I do not know if spring-cloud-stream generates the same id for the same message if it is redelivered.

Also keep in mind that even in the newer versions of spring-cloud-stream, if you use the deprecated @StreamListener, you will have the same problem.

IonutB
  • 71
  • 1
  • 6