How to read from multiple EventHub partitions simultaneously with high throughput?

Question

My one role instance needs to read data from 20-40 EventHub partitions at the same time (context: this is our internal virtual partitioning scheme - 20-40 partitions represent scale out unit).

In my prototype I use below code. By I get throughput 8 MBPS max. Since if I run the same console multiple times I get throughput (perfmon counter) multiplied accordingly then I think this is not neither VM network limit nor EventHub service side limit.

I wonder whether I create clients correctly here...

Thank you! Zaki

        const string EventHubName = "...";
        const string ConsumerGroupName = "...";

        var connectionStringBuilder = new ServiceBusConnectionStringBuilder();
        connectionStringBuilder.SharedAccessKeyName = "...";
        connectionStringBuilder.SharedAccessKey = "...";
        connectionStringBuilder.Endpoints.Add(new Uri("sb://....servicebus.windows.net/"));
        connectionStringBuilder.TransportType = TransportType.Amqp;

        var clientConnectionString = connectionStringBuilder.ToString();
        var eventHubClient = EventHubClient.CreateFromConnectionString(clientConnectionString, EventHubName);

        var runtimeInformation = await eventHubClient.GetRuntimeInformationAsync().ConfigureAwait(false);

        var consumerGroup = eventHubClient.GetConsumerGroup(ConsumerGroupName);

        var offStart = DateTime.UtcNow.AddMinutes(-10);
        var offEnd = DateTime.UtcNow.AddMinutes(-8);

        var workUnitManager = new WorkUnitManager(runtimeInformation.PartitionCount);

        var readers = new List<PartitionReader>();
        for (int i = 0; i < runtimeInformation.PartitionCount; i++)
        {
            var reader = new PartitionReader(
                consumerGroup,
                runtimeInformation.PartitionIds[i],
                i,
                offStart,
                offEnd,
                workUnitManager);
            readers.Add(reader);
        }

    internal async Task Read()
    {
        try
        {
            Console.WriteLine("Creating a receiver for '{0}' with offset {1}", this.partitionId, this.startOffset);
            EventHubReceiver receiver = await this.consumerGroup.CreateReceiverAsync(this.partitionId, this.startOffset).ConfigureAwait(false);
            Console.WriteLine("Receiver for '{0}' has been created.", this.partitionId);

            var stopWatch = new Stopwatch();
            stopWatch.Start();

            while (true)
            {
                var message =
                    (await receiver.ReceiveAsync(1, TimeSpan.FromSeconds(10)).ConfigureAwait(false)).FirstOrDefault();
                if (message == null)
                {
                    continue;
                }

                if (message.EnqueuedTimeUtc >= this.endOffset)
                {
                    break;
                }

                this.processor.Push(this.partitionIndex, message);
            }

            this.Duration = TimeSpan.FromMilliseconds(stopWatch.ElapsedMilliseconds);
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex);
            throw;
        }
    }

Answer 1

The above code snippet you provided is effectively: creating 1 Connection to ServiceBus Service and then running all receivers on one single connection (at protocl level, essentially, creating multiple Amqp Links on that same connection).

Alternately - to achieve high throughput for receive operations, You will need to create multiple connections and map your receivers to connection ratio to fine-tune your throughput. That's what happens when you run the above code in multiple processes.

Here's how:

You will need to go one layer down the .Net client SDK API and code at MessagingFactory level - you can start with 1 MessagingFactory per EventHubClient. MessagingFactory is the one - which represents 1 Connection to EventHubs service. Code to create a dedicated connection per EventHubClient:

var connStr = new ServiceBusConnectionStringBuilder("Endpoint=sb://servicebusnamespacename.servicebus.windows.net/;SharedAccessKeyName=saskeyname;SharedAccessKey=sakKey");
connStr.TransportType = TransportType.Amqp;
var msgFactory = MessagingFactory.CreateFromConnectionString(connStr.ToString());
var ehClient = msgFactory.CreateEventHubClient("teststream");

• I just added connStr in my sample to emphasize assigning TransportType to Amqp.

You will end up with multiple connections with outgoing port 5671:

If you rewrite your code with 1 MessagingFactory per EventHubClient (or a reasonable ratio) - you are all set (in your code - you will need to move EventHubClient creation to Reader)!

The only extra criteria one need to consider while creating multiple connections is the Bill - only 100 connections are included (including senders and receivers) in basic sku. I guess you are already on standard (as you have >1 TUs) - which gives 1000 connections included in the package - so no need to worry - but mentioning just-in-case.

~Sree

Answer 2

A good option is to create a Task for each partition. This a copy of my implementation which is able to process a rate of 2.5k messages per second per partition. This rate will be also related to your downstream speed.

    static void EventReceiver()
    {
        for (int i = 0; i <= EventHubPartitionCount; i++)
        {
            Task.Factory.StartNew((state) =>
            {
                Console.WriteLine("Starting worker to process partition: {0}", state);

                var factory = MessagingFactory.Create(ServiceBusEnvironment.CreateServiceUri("sb", "tests-eventhub", ""), new MessagingFactorySettings()
                {
                    TokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider("Listen", "PGSVA7L="),
                    TransportType = TransportType.Amqp
                });

                var client = factory.CreateEventHubClient("eventHubName");
                var group = client.GetConsumerGroup("customConsumer");

                Console.WriteLine("Group: {0}", group.GroupName);

                var receiver = group.CreateReceiver(state.ToString(), DateTime.Now);


                while (true)
                {
                    if (cts.IsCancellationRequested)
                    {
                        receiver.Close();
                        break;
                    }

                    var messages = receiver.Receive(20);
                    messages.ToList().ForEach(aMessage =>
                    {                            
                         // Process your event
                    });

                    Console.WriteLine(counter);
                }
            }, i);
        }
    }