4

For twitter source it is creating a 30 bit header for each kafka messages if we use custom kafka consumer other than @StreamListener (Sink). @StreamListener sends the messages correctly to the annotated method (no header). The problem is for each source the header size varies. So how can I dynamically understand the header size of the messages since I'm using custom kafka consumer.? Or How can I remove the header for each message.?

UPDATE:-

Please find below the message received in kafka-console-consumer while using a TCP Source.

�
  contentType"application/octet-stream"hello world

I just need to get the message only. In this case, hello world

Devas
  • 1,544
  • 4
  • 23
  • 28

3 Answers3

2

What version of Kafka you are using? The reason why I am asking is that starting 0.11 Kafka supports native headers and you would not have this problem anymore.

Oleg Zhurakousky
  • 5,820
  • 16
  • 17
  • Note that that requires overriding the Spring Cloud Stream binder in the pom to the kafka11 artifact (if you are using Ditmars). For other binders, set `spring.cloud.stream.binding.output.header-mode=raw`. See [the documentation about `headerMode`](https://docs.spring.io/spring-cloud-stream/docs/Chelsea.SR2/reference/htmlsingle/index.html#_producer_properties). – Gary Russell Nov 29 '17 at 13:46
  • Thanks for the support. My kafka version is 0.10. Do you mean that if I use 0.11 kafka, then the header send by the Source won't come in the kafka message. Currently I'm testing the same using a kafka-console-consumer. I have updated the message received through the console consumer. – Devas Nov 30 '17 at 08:36
  • 2
    As I said, set headermode=raw to remove the embedded header. – Gary Russell Dec 04 '17 at 16:13
  • Do I need to upgrade to 0.11 kafka to work this, because I tried with my .10 but the header was still there. Also do I need to build the artifacts with kafka 11.? – Devas Dec 04 '17 at 16:56
  • 1
    It got worked. When we set the following property on the source app, the kafka consumer doesn't receives the header. `spring.cloud.stream.default.producer.headerMode=raw`. A slight change in the property suggested by @Gary Russell. Also the same is working for kafka 0.10. – Devas Dec 08 '17 at 12:03
  • 1
    Sorry, yes, I missed the `.producer.` part. – Gary Russell Dec 08 '17 at 17:24
1

Setting spring.cloud.stream.default.producer.headerMode=raw in source will remove the header while reading the message in consumer.

Devas
  • 1,544
  • 4
  • 23
  • 28
0

Could also change this setting to remove the annoying header

spring.cloud.stream.default.producer.useNativeEncoding=true
zt1983811
  • 1,011
  • 3
  • 14
  • 34