Directing messages to consumers

Question

My client is attempting to send messages to the receiver. However I noticed that the receiver sometimes does not receive all the messages sent by the client thus missing a few messages (not sure where the problem is ? Client or the receiver). Any suggestions on why that might be happening. This is what I am currently doing

On the receiver side this is what I am doing.

This is the Event Processor

        async Task IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
        {
            foreach (var eventData in messages)
            {
                var data = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
            }
        }

This is how the client connects to the event hub

var StrBuilder = new EventHubsConnectionStringBuilder(eventHubConnectionString)
{
 EntityPath = eventHubName,
};
this.eventHubClient = EventHubClient.CreateFromConnectionString(StrBuilder.ToString());

How do I direct my messages to specific consumers

Can you provide the completed code of sending client? in the send client, how do you call the customized method Send(string content)?And in the receiver side, I see you call CheckpointAsync() in CloseAsync() method, but in official doc, the CheckpointAsync() is set in ProcessEventsAsync() method. If you don't have some custom requirements, you should follow the [official sample](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-dotnet-standard-getstarted-send#send-events) for send/receive. — Ivan Glasenberg, Nov 25 '19 at 08:38
@IvanYang The user calls SendMessage which calls Send. I highly doubt misplacing CheckpointAsync() in CloseAsync() is responsible since CloseAsync() doesnt get called while the messges are being sent. — MistyD, Nov 25 '19 at 08:47
can you share AsyncDispatchEvent method? I just tried, it cannot repro here. — Ivan Glasenberg, Nov 25 '19 at 09:34
I see the changes. And here is one thing I want to confirm, the SendMessage() method only sends one data each time. So when user calls it, the user use loop(like while or for loop) to include the SendMessage(), or just send one data each time? — Ivan Glasenberg, Nov 25 '19 at 09:41
The send message is never called from inside a for loop. Let me know if that helps — MistyD, Nov 25 '19 at 09:46
You mean it directly calls SendMessage() method and just sends one message each time? If that's the case, I will follow your code to debug it. — Ivan Glasenberg, Nov 25 '19 at 09:49
@IvanYang Seems like the messages I get are all from a certain partition.Does the method` ProcessEventsAsync` listen on all partitions ? — MistyD, Nov 26 '19 at 12:34
Yes, it should listen on all partitions. It's a little weird, I cannot repro it. I need some time to look into it. — Ivan Glasenberg, Nov 27 '19 at 00:25
And can you provide your completed code for send and receive? It seems that the code in the post is a little mess-up. I cannot totally follow up your code:(. — Ivan Glasenberg, Nov 27 '19 at 06:49
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/203156/discussion-between-ivan-yang-and-mistyd). — Ivan Glasenberg, Nov 27 '19 at 08:04

score 2 · Answer 1 · answered Nov 27 '19 at 02:05

This happens only when there are 2 or more Event Processor Host reading from same consumer group.

If you have event hub with 32 partitions and 2 event processor host reading from same consumer group. Then each event processor host will read from 16 partition and so on.

Similarly if 4 Event processor host parallelly reading from same consumer group then each will read from 8 partitions.

Check if you have 2 or more event processor host running on same consumer group.

Ivan Glasenberg · Accepted Answer · 2019-11-28T07:50:31.760

I'm using this sample code from eventhub official doc, for sending and receiving.

And I have 2 consumer groups: $Default and newcg. Suppose you have 2 clients, the client_1 are using the default consumer group($Default), and client_2 are using the other consumer group(newcg)

First, after create the send client, in the SendMessagesToEventHub method, we need to add a property with value. The value should be the consumer group name. Sample code like below:

    private static async Task SendMessagesToEventHub(int numMessagesToSend)
    {
        for (var i = 0; i < numMessagesToSend; i++)
        {
            try
            {
                var message = "444 Message";
                Console.WriteLine($"Sending message: {message}");
                EventData mydata = new EventData(Encoding.UTF8.GetBytes(message));

                //here, we add a property named "cg", it's value is the consumer group. By setting this property, then we can read this message via this specified consumer group.
                mydata.Properties.Add("cg", "newcg");

                await eventHubClient.SendAsync(mydata);

            }
            catch (Exception exception)
            {
                Console.WriteLine($"{DateTime.Now} > Exception: {exception.Message}");
            }

            await Task.Delay(10);
        }

        Console.WriteLine($"{numMessagesToSend} messages sent.");
    }

Then in the client_1, after create the receiver project, which use the default consumer group($Default) -> in the SimpleEventProcessor class -> ProcessEventsAsync method, we can filter out the unnecessary event data. Sample code for ProcessEventsAsync method:

        public Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
        {
            foreach (var eventData in messages)
            {
                //filter the data here
                if (eventData.Properties["cg"].ToString() == "$Default")
                {                    
                    var data = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);

                    Console.WriteLine($"Message received. Partition: '{context.PartitionId}', Data: '{data}'");
                    Console.WriteLine(context.ConsumerGroupName);
                }
            }

            return context.CheckpointAsync();
        }

And in another client, like client_2, which use another consumer group, like it's name is newcg, we can follow the steps in client_1, just a little changes in ProcessEventsAsync method, like below:

            public Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
            {
                foreach (var eventData in messages)
                {
                    //filter the data here, using another consumer group name
                    if (eventData.Properties["cg"].ToString() == "newcg")
                    {  
                       //other code
                    }
                   }

                 return context.CheckpointAsync();
               }

Yeah looks like I needed consumers. Thank you for your suggestion. — MistyD, Nov 29 '19 at 04:32

score 0 · Answer 3 · answered Nov 27 '19 at 08:42

I have tested your code and slightly modified it(different overload of EventProcessorHost constructor, and added CheckpointAsync after consuming the messages), and then did some tests.

By using the default implementation and default EventProcessorOptions(EventProcessorOptions.DefaultOptions) I can say that I did experience some latency when it comes to consuming messages, but all messages were processed successfully. So, sometimes it seems like I am not getting the messages from the certain partition, but after a certain period of time, all messages arrive:

Here you can find the actual modified code that worked for me. It is a simple console app that prints to the console if something arrives.

        string processorHostName = Guid.NewGuid().ToString();
        var Options = new EventProcessorOptions()
        {
            MaxBatchSize = 1, //not required to make it working, just for testing
        };
        Options.SetExceptionHandler((ex) =>
        {
            System.Diagnostics.Debug.WriteLine($"Exception : {ex}");
        });
        var eventHubCS = "event hub connection string";
        var storageCS = "storage connection string";
        var containerName = "test";
        var eventHubname = "test2";
        EventProcessorHost eventProcessorHost = new EventProcessorHost(eventHubname, "$Default", eventHubCS, storageCS, containerName);
        eventProcessorHost.RegisterEventProcessorAsync<MyEventProcessor>(Options).Wait();

For sending the messages to the event hub and testing I used this message publisher app.

Directing messages to consumers

3 Answers3

Linked