push multiple Line Text as one message in a kafka topic

Question

I want to push a text consisting of multiple lines as one message into a kafka topic.

After I enter:

kafka-console-producer --broker-list localhost:9092 --topic myTopic

and copy my text:

My Text consists of:
two lines instead of one

I get two messages in the kafka topic, but I want to have just one. Any ideas how to achieve that? Thanks

score 10 · Accepted Answer · answered Sep 04 '18 at 09:22

You can use kafkacat for this, with its -D operator to specify a custom message delimiter (in this example /):

kafkacat -b kafka:29092 \
        -t test_topic_01 \
        -D/ \
        -P <<EOF
this is a string message 
with a line break/this is 
another message with two 
line breaks!
EOF

Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140

Resulting messages, inspected also using kafkacat:

$ kafkacat -b kafka:29092 -C \
         -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\tOffset: %o\n--\n' \
         -t test_topic_01

Key (-1 bytes):
Value (43 bytes): this is a string message
with a line break
Partition: 0    Offset: 0
--

Key (-1 bytes):
Value (48 bytes): this is
another message with two
line breaks!

Partition: 0    Offset: 1
--
% Reached end of topic test_topic_01 [0] at offset 2

Inspecting using kafka-console-consumer:

$ kafka-console-consumer \
    --bootstrap-server kafka:29092 \
    --topic test_topic_01 \
    --from-beginning

this is a string message
with a line break
this is
another message with two
line breaks!

(thus illustrating why kafkacat is nicer to work with than kafka-console-consumer because of its optional verbosity :) )

score 7 · Answer 2 · answered Sep 03 '18 at 21:27

7

It's not possible with kafka-console-producer as it uses a Java Scanner object that's newline delimited.

You would need to do it via your own producer code

answered Sep 03 '18 at 21:27

OneCricketeer

179,855
19
132
245

score 1 · Answer 3 · answered Sep 04 '18 at 04:57

1

With Console-consumer you are obviously running tests for your expected data coming from client. If it is a single message, better keep it as a single string by adding a unique delimiter as identifier. e.g.

{this is line one ^^ this is line two}

Then handle the message accordingly in your consumer job. Even if client is planning to send multiple sentences in message, better make it in a single string, it will improve serialization of your message and will be more efficient after serialization.

answered Sep 04 '18 at 04:57

AbhishekN

368
4
8

It's better not to rely on random delimiters within messages at all, though – OneCricketeer Sep 04 '18 at 11:09
Agree, in fact if test case contains having multiple lines, use "\n" delimiter to simulate this scenario. e.g. {this is line one \n this is line two} – AbhishekN Sep 04 '18 at 14:06

push multiple Line Text as one message in a kafka topic

3 Answers3