How to send only custom Header using spring cloud stream

Question

I have to set a custom header in the kafka message, my kafka cluster natively supports headers (1.x.x). I am currently using springCloudVersion=Finchley.RELEASE.

When I am setting property

default:
    producer:
      headerMode: none

None of the headers are coming in the output.

On the other hand, If I am setting

default:
    producer:
      headerMode: headers

Many headers including contentType and spring_json_header_types are coming in the header which greatly affects the throughput. Kafka being a language/framework agnostic message delivery mechanism, I feel, spring should provide a way to only include user-provided headers. Is there any work around to get only user-set headers to the kafka topic while suppressing all the spring cloud related headers.

Stefano Curcio · Answer 1 · 2022-03-22T13:34:27.123

I was able to remove the "spring_json_header_types" header:

extending DefaultKafkaHeaderMapper:

package com.config.streams;    
import org.apache.kafka.common.header.Headers;
import org.springframework.kafka.support.DefaultKafkaHeaderMapper;
import org.springframework.messaging.MessageHeaders;

public class KafkaHeaderMapperNoSpringHeader extends DefaultKafkaHeaderMapper {

    @Override
    public void fromHeaders(MessageHeaders headers, Headers target) {
        super.fromHeaders(headers, target);
        target.remove("spring_json_header_types");
    }

}

instantiating the bean:

@Configuration
public class StreamsConfiguration {

       @Bean
       public KafkaHeaderMapper kafkaHeaderMapperNoSpringHeader() {
           return new KafkaHeaderMapperNoSpringHeader();
       } ...

and adding the following property in the application.properties:

spring.cloud.stream.kafka.binder.headerMapperBeanName = kafkaHeaderMapperNoSpringHeader

score 0 · Accepted Answer · edited Jun 20 '20 at 09:12

0

I already answered this on GitHub. It's a waste of your time and ours to ask the same question in multiple places. As I said there, Stack Overflow is the preferred place to ask questions.

Here's the answer again:

Please use Stack Overflow to ask questions; GitHub issues are for reporting bugs or asking for new features.

You can provide a custom header mapper in the binder configuration.

spring.cloud.stream.kafka.binder.headerMapperBeanName

The bean name of a KafkaHeaderMapper used for mapping spring-messaging headers to and from Kafka headers. Use this, for example, if you wish to customize the trusted packages in a DefaultKafkaHeaderMapper that uses JSON deserialization for the headers.

Default: none.

Implement your own KafkaHeaderMapper and add it as a @Bean and configure the binder to use it.

EDIT

The DefaultKafkaHeaderMapper has a property to enable converting byte[] headers to String.

/**
 * Set the headers to not perform any conversion on (except {@code String} to
 * {@code byte[]} for outbound). Inbound headers that match will be mapped as
 * {@code byte[]} unless the corresponding boolean in the map value is true,
 * in which case it will be mapped as a String.
 * @param rawMappedHeaders the header names to not convert and
 * @since 2.2.5
 * @see #setCharset(Charset)
 * @see #setMapAllStringsOut(boolean)
 */
public void setRawMappedHeaders(Map<String, Boolean> rawMappedHeaders) {

edited Jun 20 '20 at 09:12

Community

1
1

answered Oct 29 '19 at 21:32

Gary Russell

166,535
14
146
179

1

Thanks Gary, this worked like a charm!. On a different note, Posted the question in SO and Github with different intention, to get attention from 2 different userbase. While I agree that some users like yourself is overlapping across both platforms. From SO, I was expecting a work around from a fellow developer. From GitHub, I was expecting more authentic answer or initiate a conversation for making the platform better or a potential contribution. That being said, I should have made my intention clear. Sorry – so-random-dude Oct 30 '19 at 00:19
1

That's a fair point. We just get a bit irked when we get the same question in multiple places - some people also ask on Gitter too! For general questions how do I do this or that, we prefer Gitter for simple questions; when code/config is needed we prefer SO; with GitHub, if you had phrased it more like. "Enhancement request - I need a mechanism for more control over header mapping", I wouldn't have reacted the same way and would have replied with the mechanism that already exists. Glad it worked out for you. – Gary Russell Oct 30 '19 at 00:34
Its all good Gary. Thanks for your contributions to the opensource world. BTW, I ended up extending your own DefaultKafkaHeaderMapper and just overrode fromHeaders() method – so-random-dude Oct 30 '19 at 03:17
any code example for this on how to consume a Kafka Header of and convert to ? – AlexCon Mar 16 '20 at 19:47
It's not clear what you mean and you shouldn't ask new questions in comments on old answers; ask a new question and provide more details of what you are trying to do. – Gary Russell Mar 16 '20 at 19:51
I wanted to know if there is an implementation for what you just described above. Any Java code example on how to read Headers that are String, byte[] and convert them to String, String for Spring Cloud Stream Kafka to consume. Right now when I pick a message up, the value is of type byte[] – AlexCon Mar 16 '20 at 20:01
You can add the desired header name to the `rawMappedHeaders` Map property with the value `Boolean.TRUE` which means convert a header with that name to a String using a charset (defaults to UTF-8). – Gary Russell Mar 16 '20 at 20:09
Thanks Gary. Can I do this via application.yml or should I create a bean and inject? – AlexCon Mar 16 '20 at 21:36
The mapper has to be defined as a bean and then you put the bean name in the yml `spring.cloud.stream.kafka.binder.headerMapperBeanName`. We have an [open issue](https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/585) to add it to the yml. – Gary Russell Mar 16 '20 at 21:40

How to send only custom Header using spring cloud stream

2 Answers2