kcat protobuf deserialization

Question

I'm using kcat to check the content of kafka topics when working locally but, when messages are serialized with protobuf, the result I get is an unreadable stream of encoded characters. I'm aware of the existence of some other kafka-consumers tools (Kafdrop, AKHQ, Kowl, Kadek...) but I'm looking for the simplest option which fits my needs.

Does kcat support protobuf key/value deserialization from protofile?
Is there any simple terminal-based tool which allows this?

score 1 · Accepted Answer · answered Nov 09 '22 at 21:43

1

I've had luck with this command:

% kcat -C -t <topic> -b <kafkahost>:9092 -o -1 -e -q -D "" | protoc --decode=<full message class> path/to/my.proto --proto_path <proto_parent_folder>

answered Nov 09 '22 at 21:43

James

270
1
10

And, following your sample command, if `my.proto` has dependencies with other protofiles these dependencies would be resolved as long as they are under `` right? – frblazquez Jan 24 '23 at 11:06
@frblazquez I believe so – James Jan 24 '23 at 17:14

score 0 · Answer 2 · answered Nov 03 '22 at 22:37

any simple terminal-based tool which allows this

Only ones that integrate with the Confluent Schema Registry (which is what those linked tools use as well), e.g. kafka-protobuf-console-consumer is already part of Confluent Platform.

Regarding kcat - refer https://github.com/edenhill/kcat/issues/72 and linked issues

kcat protobuf deserialization

2 Answers2