3

When I stand up a new Azure Function listening to our event hub, with a new consumer group, it gets all the messages that have ever been sent to the event hub, even from months ago somehow. I thought setting the message retention on the hub to one day would limit that, but apparently not. Anyway, I really only want the new Azure Function to receive events from the initial deployment or testing point onward, however I don't know how I can specify that initial starting checkpoint, since that stuff is magically done in storage by AF.

If I'm doing the hub logic manually I can specify this checkpoint and start from an arbitrary time, such as the current time. Is there a way to replicate this with an Azure Function?

Josh
  • 6,944
  • 8
  • 41
  • 64

1 Answers1

7

The good news is Yes, you can change the checkpoint. There is not much magic there: the checkpoints are stored in Blob Storage -> azure-webjobs-eventhubs container in blobs with names like yournamespace.servicebus.windows.net/yourhub/$Default/X, where $Default is consumer group name and X is partition number.

Here is an example of such blob:

{"PartitionId":"0","Owner":null,"Token":null,"Epoch":567,"Offset":"14992",
 "SequenceNumber":156}

The bad news is that you need to know your offset for the moment you want to start at, and you probably don't know it. And then, you need to go ahead and modify the blobs either manually or with custom code. And you should do that before the Function App starts.

It's up to you to decide whether you can/want to do that. AFAIK, there's no easy and official way to accomplish this.

Mikhail Shilkov
  • 34,128
  • 3
  • 68
  • 107
  • I discovered this. I took another function and basically copied the Epoch, Offset and SequenceNumber but it still seemed to process from the start (or way more messages than it should). – Josh Nov 29 '17 at 22:09
  • @Josh After the function started working, did it change your values? Were they growing, or went down initially? – Mikhail Shilkov Nov 29 '17 at 22:12
  • It did change them, all three values increased. Epoch by 1, Offset by about 10k, SequenceNumber by about 40. – Josh Nov 29 '17 at 22:38
  • I was able to get this working by copying an existing set of checkpoint files and changing Token and Owner to null, without touching the numbers. – Josh Dec 04 '17 at 21:03
  • I just deleted them and reprocessed everything. It worked, thanks! – Ana Franco Jun 17 '21 at 20:13