0

I have been trying to use kafkacat to find a message in a topic and publish it back into the topic. We use protobuf so the message values should be in bytes (Keys can be different such as strings or bytes). However, I am unable to publish the message that could be deserialized properly.

How can I do this with kafkacat? I am also open to using other recommended tools for doing this.

Example attempt:

kafkacat -b <broker> -t <topic> -o -10 -e -c 1 -C -K: > test2.txt

cat test2.txt | kafkacat -b <broker> -t <topic> -P -K:

test2.txt shows:

21aa7e2f-41a1-4972-9108-3057627d53f0:
Y/<protobuf.class.path>i
$21aa7e2f-41a1-4972-9108-3057627d53f0A
DIABETEBG_METERBG300"BG300*QP3687WK02000012????

But when I use the same kafkacat consumer command to get the result, I just get the last line:

DIABETEBG_METERBG300"BG300*QP3687WK02000012????

I think the problem is that the consuming outputs lines (might be a part of the payload?) And the producer is treating each line as a new message.

Thomas Freiling
  • 67
  • 1
  • 8

1 Answers1

0

the producer is treating each line as a new message

That's correct.

If you have a binary file, I suggest writing code for this, as kafkacat assumes UTF8 encoded strings as input

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I'm hoping for some standard tooling that does this, as I think this is a pretty common use case. Also, I would like a tool like kafka-console-consumer/producer or kafkacat with all of its options available. like selecting a particular partition/offset, or searching by timestamp, etc.. – Thomas Freiling Feb 23 '21 at 21:24
  • 1
    I ended up writing a script. – Thomas Freiling Feb 24 '21 at 06:00
  • Sending whole files as a Kafka message is not a common use case – OneCricketeer Feb 24 '21 at 14:32
  • 1
    Agreed, but simply sending messages from one kafka topic to another is though. I ended up writing a script in scala using apache kafka libraries (and cakesolutions) that just used Bytes (de)serialization). I added as input a list of headers of the original message to keep (which was needed for our use case) – Thomas Freiling Feb 24 '21 at 19:08
  • From one topic to another, built in tool would be MirrorMaker – OneCricketeer Mar 01 '21 at 15:28