How to get Azure EventHub Depth

Question

My EventHub has millions of messages ingestion every day. I'm processing those messages from Azure Function and printing offset and squence number value in logs.

public static async Task Run([EventHubTrigger("%EventHub%", Connection = "EventHubConnection", ConsumerGroup = "%EventHubConsumerGroup%")]EventData eventMessage,
        [Inject]ITsfService tsfService, [Inject]ILog log)
    {
log.Info($"PartitionKey {eventMessage.PartitionKey}, Offset {eventMessage.Offset} and SequenceNumber {eventMessage.SequenceNumber}");
}

Log output

PartitionKey , Offset 78048157161248 and SequenceNumber 442995283

Questions

PartitionKey value blank? I have 2 partitions in that EventHub
Is there any way to check backlogs? Some point of time I want to get how many messages my function need to process.

score 3 · Accepted Answer · answered Jan 23 '19 at 15:48

Yes, you can include the PartitionContext object as part of the signature, which will give you some additional information,

public static async Task Run([EventHubTrigger("HubName", 
    Connection = "EventHubConnectionStringSettingName", 
    ConsumerGroup = "Consumer-Group-If-Applicable")] EventData[] messageBatch, PartitionContext partitionContext, ILogger log)

Edit your host.json and set enableReceiverRuntimeMetric to true, e.g.

"version":  "2.0",
"extensions": {
    "eventHubs": {
        "batchCheckpointFrequency": 100,
        "eventProcessorOptions": {
            "maxBatchSize": 256,
            "prefetchCount": 512,
            "enableReceiverRuntimeMetric": true
        }            
    }
}

You now get access to RuntimeInformation on the PartitionContext, which has some information about the LastSequenceNumber, and your current message has it's own sequence number, so you could use the difference between these to calculate a metric, e.g something like,

public class EventStreamBacklogTracing
{
    private static readonly Metric PartitionSequenceMetric = 
        InsightsClient.Instance.GetMetric("PartitionSequenceDifference", "PartitionId", "ConsumerGroupName", "EventHubPath");

    public static void LogSequenceDifference(EventData message, PartitionContext context)
    {
        var messageSequence = message.SystemProperties.SequenceNumber;
        var lastEnqueuedSequence = context.RuntimeInformation.LastSequenceNumber;

        var sequenceDifference = lastEnqueuedSequence - messageSequence;

        PartitionSequenceMetric.TrackValue(sequenceDifference, context.PartitionId, context.ConsumerGroupName,
            context.EventHubPath);
    }
}

I wrote an article on medium that goes into a bit more detail and show how you might consume the data in grafana,

https://medium.com/@dylanm_asos/azure-functions-event-hub-processing-8a3f39d2cd0f

Sorry, not sure about V1 - I don't think you have access to the enableReceiverRuntimeMetric settings, which seems to be V2 only - https://github.com/Azure/Azure-Functions/pull/1103/files. However, the technique should be the same - to calculate how many messages you have to process, you need to determine where you are relative to the end of the stream. — Dylan Morley, Jan 24 '19 at 09:11

score 0 · Answer 2 · answered Apr 29 '20 at 14:59

PartitionKey value blank? I have 2 partitions in that EventHub

The partition key is not the same as the partition ids. When you publish an event to Event Hubs, you can set the partition key. If that partition key is not set, then it will be null when you go to consume it.

Partition key is for events where you don't care what partition it ends up in, just that you want events with the same key to end up in the same partition.

An example would be if you had hundreds of IoT devices transmitting telemetry data. You don't care what partition these IoT devices publish their data to, as long as it always ends up in the same partition. You may set the partition key to the serial number of the IoT device. When that device publishes its event data with that key, the Event Hubs service will calculate a hash for that partition key, map it to a specific Event Hub partition, and will route any events with that key to the same partition.

The documentation from "Event Hubs Features: Publishing an Event" depicts it pretty well.

How to get Azure EventHub Depth

2 Answers2