0

According too Microsoft docs

There can be at most 5 concurrent readers on a partition per consumer group; however it is recommended that there is only one active receiver on a partition per consumer group. Within a single partition, each reader receives all of the messages. If you have multiple readers on the same partition, then you process duplicate messages. You need to handle this in your code, which may not be trivial. However, it's a valid approach in some scenarios.

  1. How do we configure our EventHubConsumerClient's to load balance partitions equally across consumers in the same consumer group?

  2. How can we prevent two or more consumers in the same consumer group, consuming from the same partition?

Note: using node azure eventhubs sdk https://www.npmjs.com/package/@azure/event-hubs

Thanks in advance!

Jessica
  • 167
  • 2
Kevin Peters
  • 1
  • 1
  • You would manage that manually on the consumer side. If you look at the picture in the linked documentation, everything to the right of the hub is your responsibility to configure correctly. The complexity comes in the flexibility of the hub – Patrick Goode Apr 16 '20 at 23:30
  • from last one year, I'm reading millions of data every day from Eventhub. I have multiple Azure function and each function has own consumer group and I never face duplicate data issue. I'm using .net Azure function – Pankaj Rawat Apr 17 '20 at 05:10

1 Answers1

1

If you're creating your EventHubConsumerClient using the overload that accepts a checkpoint store, then the client will manage load balancing in a cooperative manner and will ensure that a partition is being read by a single client.

This sample discusses your scenario and provides a snippet that illustrates the basic flow.

The one thing that I'll note, however, is that even when the EventHubConsumerClient is load balancing, your application still must be idempotent in the face of duplicate events being read/processed. Event Hubs offers an at-least-once guarantee; in rare cases it is possible for the service to return an event more than once.

Jesse Squire
  • 6,107
  • 1
  • 27
  • 30