2

I want to do compress data log in Apache Kafka. How do I know which one to choose? For me, performance and space is important.

Server.properties file

Compression.type = snappy , gzip , lz4 vb. use.

omerstack
  • 535
  • 9
  • 23
  • If I understand correctly, it's not really log compression you're looking for but message compression. Can you please change the title if my understanding is correct? – Yassin Hajaj Dec 19 '20 at 19:58

1 Answers1

2

Anecdotally, Uber uses zlib with MsgPack serialized messages. However, you should perform your own benchmarks on your own hardware, network and storage (for example, those numbers were gotten using Python libraries)

Regarding the underlying serialization, Avro serialization via a Schema Registry allows you to have stricter schema definition rules than plaintext or JSON, and Avro generally pairs well with Snappy compression

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245