3

So I am using EventHubs for data pipeline queuing, but in order to get the benefit of buffered ingestion, I need to know how many items are in a partition at any one time. Depending on the consumer group and partition id this can give massive insight into which receivers need to be scaled or aren't performing as expected.

I am aware you can create alerts based on the number of messages, but what I really want is the ability to request the current item count of a given Namespace, EventHub, Consumer Group or Partition (whatever is available) from within my C# code.

I can see that you can easily get the queue size in bytes using the .NET ServiceBus API, but I can't see any way of getting the item count?

tommed
  • 1,521
  • 2
  • 19
  • 33
  • It sounds like what you are actually asking for is a way to tell the consumer lag of the various partitions (or at least this is what one generally needs). That is "how many items is a given consumer behind in each partition". I'd suggest asking another question around that, though I haven't written that very necessary code yet. It's a common thing to monitor with Apache Kafka. – cacsar May 23 '16 at 08:54

1 Answers1

2

According to the Event Hubs REST API, there is no way to get the amount of messages in the Event Hub for an exact moment. You may try to play with the Incoming Messages amount and Outgoing Messages amount or get the size of the messages in the Hub, if you know at least approx. the size (and divide by that).

Alex Belotserkovskiy
  • 4,012
  • 1
  • 13
  • 10
  • Strange, I watched a video from Build 2016 last week where they were selling up Event Hubs for buffered ingestion, stating you could monitor the size of the queues - the file size of the queue doesn't really give you an indication of item size unless you have consistent payload sizes :-/ – tommed May 20 '16 at 16:44
  • 2
    Quote: "However, unlike queues, there is no concept of queue length in Azure Event Hubs because the data is processed as a stream". https://blogs.msdn.microsoft.com/kaevans/2015/02/24/scaling-azure-event-hubs-processing-with-worker-roles/ – tommed May 22 '16 at 11:56