0

I am curious about how compression works in pulsar. from the public doc, it states "You can compress messages published by producers during transportation" does it mean client compress the data and the data get decompressed when it arrives at broker so the decompressed data is persisted and consumer later? or it means the compression happens from end-to-end and the decompression happens at consumer side?

dashenswen
  • 540
  • 2
  • 4
  • 21

1 Answers1

2

Compression and decompression is done on the client and is transparent to the broker. The message is stored compressed into the ledgers. The compression details (algo used, ...) are part of the message metadata. The same principles apply to batching and encryption.

Christophe Bornet
  • 1,062
  • 8
  • 8
  • Can you point me to any resource? I am unable to find related information in the public doc – dashenswen Dec 07 '21 at 13:03
  • There's not much doc on this. You can see that there are compression related fields in the message metadata here : https://pulsar.apache.org/docs/en/develop-binary-protocol/#message-metadata . – Christophe Bornet Dec 07 '21 at 21:43
  • In the client source code, the payload is compressed here : https://github.com/apache/pulsar/blob/42469de02172324f974947c4b5186e897d430e09/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageContainerImpl.java#L134 and uncompressed here : https://github.com/apache/pulsar/blob/42469de02172324f974947c4b5186e897d430e09/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1190 – Christophe Bornet Dec 07 '21 at 21:43