0

I am trying to figure out an optimal event size to produce into Kafka. I may have events ranging from 1KB to 20KB and wonder if this will be an issue.

It is possible that I could make some producer changes to make them all roughly a similar size, say 1KB-3KB. Would this be an advantage or will Kafka have no issue with the variable event size?

Is there an optimal event size for Kafka or does that depend on the configured Segment settings?

Thanks.

mrmannione
  • 749
  • 10
  • 29

1 Answers1

1

By default, Kafka supports up to 1MB messages, and this can be changed to be larger, of course sacrificing network IO and latency as a result of making it larger.

That being said, I don't think it really matters if messages are consistently sized or not for the sizes of data that you are talking about.

If you really want to squeeze your payloads, you can look into different serialization frameworks and compression algorithms offered in the Kafka API.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I can keep my messages below 1MB. But my question was mainly about message size for optimal disk storage\segment or does the 1MB limit still apply? – mrmannione Dec 01 '18 at 19:11
  • The limit is per message that the broker will even accept. Again, I don't think trying to round your messages to an even factor of the segment size is a factor that needs to be taken into consideration. The segments are 1GB by default, so whether you have 1KB or 20KB, you'll be able to store thousands of events before a segment is rolled/closed – OneCricketeer Dec 01 '18 at 20:33