Is there a way to add headers in kafka-console-producer.sh

Question

I'd like to use the kafka-console-producer.sh to fire a few JSON messages with Kafka headers.

Is this possible?

docker exec -it kafka_1 /opt/kafka_2.12-2.3.0/bin/kafka-console-producer.sh --broker-list localhost:9093 --topic my-topic --producer.config /opt/kafka_2.12-2.3.0/config/my-custom.properties

Possible duplicate of [How to produce messages with headers in Kafka 0.11 using console producer?](https://stackoverflow.com/questions/49217828/how-to-produce-messages-with-headers-in-kafka-0-11-using-console-producer) — Mickael Maison, Nov 05 '19 at 17:58

score 12 · Accepted Answer · edited Feb 24 '21 at 20:39

12

No, but you can with kafkacat's -H argument:

Produce:

echo '{"col_foo":1}'|kafkacat -b localhost:9092 -t test -P -H foo=bar

Consume:

kafkacat -b localhost:9092 -t test -C -f '-----\nTopic %t[%p]\nOffset: %o\nHeaders: %h\nKey: %k\nPayload (%S bytes): %s\n'
-----
Topic test[0]
Offset: 0
Headers: foo=bar
Key:
Payload (9 bytes): col_foo:1
% Reached end of topic test [0] at offset 1

edited Feb 24 '21 at 20:39

so-random-dude

15,277
10
68
113

answered Nov 05 '19 at 19:29

Robin Moffatt

30,382
3
65
92

3

Now it's called `kcat`, just for the record :) – Yandry Pozo May 13 '22 at 17:55

score 1 · Answer 2 · answered Jul 21 '22 at 11:22

1

Starting from kafka 3.1.0 there is an option to turn on headers parsing parse.headers=true and then you just place them before your record value, info form docs:

| parse.headers=true:
|  "h1:v1,h2:v2...\tvalue"

So your command will look like

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic topic_name --property parse.headers=true

and then you pass

header_name:header_value\nrecord_value

answered Jul 21 '22 at 11:22

dswiecki

138
2
11

1

for what it's worth. The default property for `headers.delimiter` is `\t`. After some confusion. This command will need --property headers.delimiter='\n' passed along as well otherwise the CLI complains with exceptions regarding the header delimiter. – madeyejm Aug 08 '22 at 19:03
I am getting No headers delimiter found on line number 1: with this message "key:value\t sa" – saurav Sep 13 '22 at 12:16
https://cwiki.apache.org/confluence/display/KAFKA/KIP-798%3A+Add+possibility+to+write+kafka+headers+in+Kafka+Console+Producer – Srinivasu Oct 01 '22 at 06:53
Can you add one example with header as json payload and value as json payload as well. – dinesh kandpal Nov 25 '22 at 15:43

score 0 · Answer 3 · answered Jul 13 '23 at 16:00

Consumer producer command:

docker exec -it container_id kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092 --property parse.headers=true --property headers.key.separator=: --property headers.delimiter=\t

Payload format:

header_name:header_value t payload_value

Console consumer command:

docker exec -it container_id kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092 --property print.headers=true

Is there a way to add headers in kafka-console-producer.sh

3 Answers3