Sending messages to Kafka with header using KafkaProducer

Question

i am trying to create a simple utility to publish messages to Kafka, but need to pass headers as well along with the message. The utility works fine without headers but while trying to send headers i am getting error.

below is sample code that i am using -

public static void main(String[] args) throws Exception {

        Properties props = new Properties();

        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "host:port");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        KafkaProducer<String, String> producer = new KafkaProducer<>(props);

        List<Header> headers = Arrays.asList(new RecordHeader("sample_header", "sample_value".getBytes()));
        ProducerRecord<String, String> record = new ProducerRecord<>("TEST", 0, "key", "sample message", headers);
        Future<RecordMetadata> future = producer.send(record);
        System.out.println(future.get());

        producer.close();

    }

the exception i am getting is -

Exception in thread "main" java.lang.IllegalArgumentException: Magic v1 does not support record headers
    at org.apache.kafka.common.record.MemoryRecordsBuilder.appendWithOffset(MemoryRecordsBuilder.java:385)
    at org.apache.kafka.common.record.MemoryRecordsBuilder.appendWithOffset(MemoryRecordsBuilder.java:424)
    at org.apache.kafka.common.record.MemoryRecordsBuilder.append(MemoryRecordsBuilder.java:481)
    at org.apache.kafka.common.record.MemoryRecordsBuilder.append(MemoryRecordsBuilder.java:504)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.tryAppend(ProducerBatch.java:106)
    at org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:219)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:791)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:745)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:634)
    at com.test.kafka.KafkaProducerApp.main(KafkaProducerApp.java:45)

@Paizo- kafka_2.10-0.10.1.1 is the version of kafka broker .. any suggestions?? — seeker27, Jun 16 '18 at 23:00
nope sorry, maybe you are using a different library version? Here you can find some info https://stackoverflow.com/q/47953901/3224238 Why headers instead of keys? — Paizo, Jun 17 '18 at 13:34

score 1 · Answer 1 · answered Apr 04 '19 at 23:14

Had the same error through different means (Spring Cloud Streams Kafka producers). My issues was caused by having an inter.broker.protocol.version which was set to a lower version then the version of the broker itself see Kafka's broker configs it allows:

Specify which version of the inter-broker protocol will be used. This is typically bumped after all brokers were upgraded to a new version.

For my deployment the broker was v1.1 but the inter.broker.protocol.version was v0.10.1. When the v1.1 broker received a message with the headers it was fine until it went to replicate the message using v0.10.1 which doesn't support headers (producing the error of "Magic v1").

Sending messages to Kafka with header using KafkaProducer

1 Answers1