6

We are using the EventProcessorHost to receive events from Azure EventHubs. I've been unsuccessfully trying to configure it (through the EventProcessorOptions.InitialOffsetProvider) to read events from UTC now on but it always reads from the start of the feed. I am not saving checkpoints (and I even deleted the BLOB container created). This is how I am setting it:

DateTime startDate = DateTime.UtcNow;

var epo = new EventProcessorOptions
            {
                MaxBatchSize = 100, 
                PrefetchCount = 100, 
                ReceiveTimeOut = TimeSpan.FromSeconds(120),  
                InitialOffsetProvider = (name) => startDate  
            };

Any guidance would be appreciated.

Jose Parra
  • 877
  • 9
  • 23
  • I have not used EventHubs that much but I have run through the Azure Connect the Dots demos out on GitHub and in their demos they do not declare a variable like you do. They have UtcNow inline. ex: InitialOffsetProvider = (name) => DateTime.UtcNow https://github.com/Azure/connectthedots/blob/master/Azure/WebSite/ConnectTheDotsWebSite/Global.asax.cs – jdruid Nov 24 '15 at 20:11

3 Answers3

9

Think this changed in version 2.0.0 - Rajiv's code would now be:

var eventProcessorOptions = new EventProcessorOptions
{
    InitialOffsetProvider = (partitionId) => EventPosition.FromEnqueuedTime(DateTime.UtcNow)
};

Here is an example block with fully qualified classnames:

    private static async Task MainAsync(string[] args)
    {
        try{
            Console.WriteLine("Registering EventProcessor...");

            string AISEhConnectionStringEndpoint = Configuration["AISEhConnectionStringEndpoint"];
            string AISEhConnectionStringSharedAccessKeyName = Configuration["AISEhConnectionStringSharedAccessKeyName"];
            string AISEhConnectionStringSharedAccessKey = Configuration["AISEhConnectionStringSharedAccessKey"];
            string EhConnectionString = $"Endpoint={AISEhConnectionStringEndpoint};SharedAccessKeyName={AISEhConnectionStringSharedAccessKeyName};SharedAccessKey={AISEhConnectionStringSharedAccessKey}";
            string AISEhEntityPath = Configuration["AISEhEntityPath"];
            string AISEhConsumerGroupName = Configuration["AISEhConsumerGroupName"];
            string AISStorageContainerName = Configuration["AISStorageContainerName"];
            string AISStorageAccountName = Configuration["AISStorageAccountName"];
            string AISStorageAccountKey = Configuration["AISStorageAccountKey"];

            string StorageConnectionString = string.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", AISStorageAccountName, AISStorageAccountKey);

            var eventProcessorHost = new Microsoft.Azure.EventHubs.Processor.EventProcessorHost(
                AISEhEntityPath,
                AISEhConsumerGroupName,
                EhConnectionString,
                StorageConnectionString,
                AISStorageContainerName);

            var options = new Microsoft.Azure.EventHubs.Processor.EventProcessorOptions
            {
                InitialOffsetProvider = (partitionId) => Microsoft.Azure.EventHubs.EventPosition.FromEnqueuedTime(DateTime.UtcNow)
            };

            // Registers the Event Processor Host and starts receiving messages
            await eventProcessorHost.RegisterEventProcessorAsync<GetEvents>(options);

            Thread.Sleep(Timeout.Infinite);

            // Disposes of the Event Processor Host
            await eventProcessorHost.UnregisterEventProcessorAsync();
        }
        catch(Exception ex)
        {
            Console.WriteLine(ex.Message);
            NLog.LogManager.GetCurrentClassLogger().Error(ex);
            throw;
        }
    }

}

And here are my general settings with secrets/exact addresses obscured to help work things out, as I found working this out to be less pleasurable than extracting teeth:

"AISEhConnectionStringEndpoint": "sb://<my bus address>.servicebus.windows.net/",
"AISEhConnectionStringSharedAccessKeyName": "<my key name>",
"AISEhConnectionStringSharedAccessKey": "<yeah nah>",
"AISEhEntityPath": "<Event Hub entity path>",
"AISEhConsumerGroupName":  "<consumer group name e.g $Default>",
"AISStorageContainerName":  "<storage container name>",
"AISStorageAccountName": "<storage account name>",
"AISStorageAccountKey": "<yeah nah>",
leighghunt
  • 416
  • 4
  • 4
  • Which using did you use to recognise EventPosition? – Charlie S Jun 06 '18 at 13:32
  • Needed to add nuget Microsoft.Azure.EventHubs and use that as namespace. Unfortuntately, this updated code still didnt work with our event hub listener receiving data for the last week or so – Charlie S Jun 06 '18 at 14:18
  • Yep - that's the right using - I'll update the comment with some additional info. I did find figuring out the magical incantation to get Event Hub to connect to be very frustrating. Let me know if you need any more info or want to run the strings sans passwords by me. – leighghunt Jun 07 '18 at 21:01
  • @leighghunt: do you also need to set a new LeaseContainerName when initializing the event processor host? https://github.com/Azure/azure-event-hubs-dotnet/issues/233#issuecomment-358147160 – mikebridge Dec 14 '18 at 23:06
  • @mikebridge, yes I think so from memory (we moved to kafka, so I don't have a working environment to test anymore). If you've already consumed some events for a given lease container name, then any changes to the initial offset in your code from that point on will have no effect, as it's no longer used. – leighghunt Feb 03 '19 at 19:48
7

I found that the checkpoint folder in the blob was still there and my app was considering this and ignoring the date I set in EventProcessorOptions. After I deleted the container it started to run as expected (taking in count the UTC date).

Jose Parra
  • 877
  • 9
  • 23
4

You can use the EventProcessorOptions class for this and provide an offset set to the desired time.

var eventProcessorOptions = new EventProcessorOptions
{
    InitialOffsetProvider = (partitionId) => DateTime.UtcNow
};

You can then use any of the RegisterEventProcessAsync overloads that accepts eventProcessorOptions.

Rajiv Nandivada
  • 73
  • 5