0

I have Windows Service written in C#, which is subscribing to Event hubs and listening to them for any messages.

The code I have followed is as follows:

public class SimpleEventProcessor : IEventProcessor
{
    public Task CloseAsync(PartitionContext context, CloseReason reason)
    {
       Console.WriteLine($"Processor Shutting Down. Partition '{context.PartitionId}', Reason: '{reason}'.");
       return Task.CompletedTask;
    }

    public Task OpenAsync(PartitionContext context)
    {
       Console.WriteLine($"SimpleEventProcessor initialized. Partition: '{context.PartitionId}'");
       return Task.CompletedTask;
     }

    public Task ProcessErrorAsync(PartitionContext context, Exception error)
    {
       Console.WriteLine($"Error on Partition: {context.PartitionId}, Error: {error.Message}");
       return Task.CompletedTask;
    }

    public Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
    {
       foreach (var eventData in messages)
       {
          var data = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
          Console.WriteLine($"Message received. Partition: '{context.PartitionId}', Data: '{data}'");
          DoSomethingWithMessage(); // typically takes 20-20 mins to finish this method.
       }
       return context.CheckpointAsync();
    }
} 

This is the sample code mentioned in this document.

Now I have 8 partitions on my event hub. So 8 partitions keep receiving messages as and when there is a new message. Whenever a message is received, a method is called DoSomethingWithMessage() which takes around 30 mins to finish as it does some calculations.

I want my code to run synchronously, meaning when one message is received, service should first complete this method execution, then receive the next message. Now what is happening is , even when the DoSomethingWithMessage() is still under execution, new messages are being received and that message is processed parallel to the first message, which is creating some calculation problems.

Is there a way I can receive messages one by one?

halfer
  • 19,824
  • 17
  • 99
  • 186
CrazyCoder
  • 2,194
  • 10
  • 44
  • 91
  • How many messages do you get per minute? If a single message takes 30 minutes to process and you want to process them one by one you might be better of using a queue. Event hub is for streaming and processing (tens of) thousands of messages per second. – Peter Bons Feb 05 '20 at 06:52
  • Also, 8 partitions, what is the reason behind this choice as messages will be processed per partition? – Peter Bons Feb 05 '20 at 06:53
  • @PeterBons We have 8 VMs where this windows service is installed. So 8 partitions. All the 8 VMs are subscribing to the same event hub. The typical messages we receive are 1-2 message every 10-15 mins. Not all of them takes 30 mins to process. It is the average time I have mentioned. – CrazyCoder Feb 05 '20 at 07:08
  • Also we are planning to change to kafka queue instead of event hubs, but that is not planned in the near future. – CrazyCoder Feb 05 '20 at 07:10
  • Ok. For now you what you can do is have al the event processor instances forward the messages to an azure storage/service bus queue and have one listener set up to process those messages. – Peter Bons Feb 05 '20 at 07:12
  • @PeterBons , is there anything I can do from the code side instead of Infrastructure side? – CrazyCoder Feb 05 '20 at 07:25
  • Not easy to do. Each instance of the eventprocessor it attached to a single partition and lives in a separate process. So synchronizing a single queue using code will be hard. – Peter Bons Feb 05 '20 at 09:53
  • By the way, do you want to process all the message in order, no matter which service send them? (like date&time send) – Peter Bons Feb 05 '20 at 13:21
  • @PeterBons , yes. I want all of the messages to process in order , one after the other. – CrazyCoder Feb 06 '20 at 06:48
  • Then I am afraid you really are confronted with an I'll decision to go for event hub as it is not a good fit for ordered message processing. It is designed for scale and throughput. You can do ordered message processing per partition but without a lot of effort that is it. If all windows services write to a specific partition instead of using the default round robin distribution you can already do ordered processing per windows service. Otherwise not. – Peter Bons Feb 06 '20 at 09:59
  • @PeterBons , Can you confirm my understanding on this. Each partition works as a separate thread and will receive messages simultaneously. But with in a partition, we can process the messages one by one , is it? – CrazyCoder Feb 06 '20 at 10:05
  • Correct. But if the messages of a given windows service are send to different partitions (which is the default) it is hard to order those if that is the requirement – Peter Bons Feb 06 '20 at 10:21
  • @PeterBons, I have another similar questions for which I have asked another question here :https://stackoverflow.com/q/60107379/8930751 as it is slightly different form my question here. Can you also refer to that question and answer please. Thanks. – CrazyCoder Feb 07 '20 at 05:06
  • You are running with the wrong service. Given long process duration and low message throughput, you should have been using Service Bus Queues, not Event Hubs. – Serkant Karaca Feb 14 '20 at 01:54

0 Answers0