Are Kafka Partition Logs really immutable? To make it really immutable, can the logs be integrated with distributed ledger?

Question

Are Kafka Partition Logs really immutable? To make it really immutable, is it possible that these partition logs be integrated with distributed ledger such as Quorum, Hyperledger Fabric or Corda?

Understand that Kafka uses Zookeeper which uses Raft consensus algorithm (something I am very familiar with). The Kafka cluster nodes are assumed to be trusted and for a good reason - speed.

For true immutability, I believe kafka partition logs can and should be integrated with a distributed ledger - Is this something within the realm of possibility? Has that been done?

Nathan Aw (Singapore)

You mean transforming the kafka partition log in a blockchain? — Diego Marin Santos, May 24 '20 at 12:39
honestly to me events should be immutable and it only makes sense to put the kafka partition log in the blockchain — Nathan Aw, May 28 '20 at 01:30

score 0 · Answer 1 · answered May 27 '20 at 20:13

If you wanted to manipulate your logs into an immutable ledger I don't know of any DLT's that support that out of the box.

In the case of Corda, you could definitely do it just by having a Corda node with cordapps that can store states and adjust them based on how you want to represent your logs. (lots of sample apps on https://github.com/corda/samples-kotlin and https://github.com/corda/samples-java)

Just try to be cognizant of transaction limits. I believe corda enterprise can currently do about 6000 transactions per second but if you're trying to store your logs within a blockchain.

Best of luck!

Are Kafka Partition Logs really immutable? To make it really immutable, can the logs be integrated with distributed ledger?

1 Answers1