30

What's the difference between the following ways of enabling compression in kafka:

Approach 1: Create a topic using the command:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --config compression.type=gzip --topic test

Approach 2: Set the property compression.type = gzip in Kafka Producer Client API.

I get better compression and higher throughput when using Approach 1.

If I use Approach 1, does it mean that the compression occurs at the broker end while in Approach 2, the messages are compressed at Producer end and then sent to broker?

shants
  • 612
  • 1
  • 7
  • 12

1 Answers1

26

If I use Approach 1, does it mean that the compression occurs at the broker end?

It depends. If the producer does not set a compression.type or sets a different one, then the message will be compressed at the broker end. However, if producer also sets compression.type to gzip, no need to compress again at the broker end. Actually, there are other strict conditions that must be met to ensure no need to compress, although it's a little bit beyond of the scope.

in Approach 2, the messages are compressed at Producer end and then sent to broker?

Yes, records will be compressed before being sent to the broker if producer sets its compression.type config.

amethystic
  • 6,821
  • 23
  • 25
  • 12
    "there are other strict conditions that must be met to ensure no need to compress" can you elaborate? I think its relevant to the question, if the broker ends up re-compressing anyway then its a waste of CPU to do this at the producer. Where can I find info about the scenarios that this happens? – dtheodor Sep 21 '18 at 10:25
  • I would also appreciate some elaboration on these "other strict conditions", even if only as a reference to the relevant source code. – Simon Lepkin Jul 03 '19 at 16:50
  • 3
    @SimonLepkin See https://stackoverflow.com/a/59902540/4594699 for more information. That information is after reading the source code and a whitepaper regarding how to tune your Kafka setup. – Ashhar Hasan Jan 28 '20 at 05:19