0

I'm looking to implement a scenario where consumer order does not matter, and want to publish to multiple partitions.

In this scenario, what would the strategy be used to select a partition if partition-key-expression is not specified in the producer definition? i.e with this config I see the messages being published to all 4 topics, but it is not clear what mechanism is being used:

spring:
  cloud:
    stream:
      bindings:
        outbound:
          destination: mytopic
          group: app
          producer:
            partitioned: true
            partition-count: 4
Funsaized
  • 1,972
  • 4
  • 21
  • 41

1 Answers1

1

Binder level partitioning is really not needed with the Kafka binder; it is intended for binders where the underlying technology does not have native partitioning.

There is no need to set the partitioned=true property.

Kafka's default strategy is used

https://kafka.apache.org/documentation/#producerconfigs_partitioner.class

If there is no partitioner.class specified...

If not set, the default partitioning logic is used. This strategy will try sticking to a partition until batch.size bytes is produced to the partition.

It works with the strategy:

If no partition is specified but a key is present, choose a partition based on a hash of the key

If no partition or key is present, choose the sticky partition that changes when batch.size bytes are produced to the partition.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • gotcha - ty! also found it under the heading "Some caveats to keep in mind" in this article https://spring.io/blog/2021/02/03/demystifying-spring-cloud-stream-producers-with-apache-kafka-partitions – Funsaized Nov 02 '22 at 18:50
  • missed it on that first read through – Funsaized Nov 02 '22 at 18:57