3

Im building a queue processing webjob to read messages from a queue and use the data to retrieve a report uri from blob storage then send this as a link in an email. I have the process working perfectly well, but I need to send the email within a specific time window.

I have another process (webjob) that retrieves this data from a sql backend and places it in the email queue.

This webjob runs every 30 mins and only gets data for the current day and within a 2 hour window of the current time. So I know that anything in the queue is for today and within 2 hours of 'now'. How would i narrow this down further to read data from the queue, and if the 'email out' time is set to say 19:00 and the current time is 18:00, I can put this message back in the queue to read it again later, next time it should be closer to 19:00 then I can process it and send it out in an email.
The time doesn't have to be spot on, so even if its within, say 30 mins of 19:00 (or whatever time its set to be sent) It can be processed. So Im effectively taking an object from the queue, checking its time and if its not within say 30 mins of its allotted 'email out' time, I place it back in the queue and its processed again

** in my webjob, I have a 'Functions' class that contains a method 'ProcessQueueMessage' which is triggered whenever a message is placed in the queue.

// This function will get triggered/executed when a new message is written 
    // on an Azure Queue called queue.
    public async Task ProcessQueueMessage([QueueTrigger("%reportgenerator%")] Data.Dto.Schedule.ScheduleDto schedule)
    {
        var reports = await this._scheduledReportGenerationService.GenerateScheduledEmails(schedule.ID);
    }

The ScheduleDto class will have a generation time property, I can read this and compare it to the current time and process it only if its within my specified 'time window'. How would i stop the queue message from being deleted here so that I can re-process it ?

proteus
  • 545
  • 2
  • 12
  • 36
  • Are you using Azure Storage Queues or Service Bus Queues? – Gaurav Mantri Aug 11 '17 at 08:04
  • 1
    Also, it is not clear from your question if you're using 2 webjobs or just 1 webjob. One minor request: Please take a moment and format your question and break it in paragraphs. Reading one big large paragraph is a pain :). – Gaurav Mantri Aug 11 '17 at 08:08
  • I have 2 webjobs, one takes data from the backend for 'today within 2 hours' , creates any reports specified and puts it in blob storage. The other reads this data, and sends the URI of the blob (pdf) out in an email. Im using Azure Storage Queues – proteus Aug 11 '17 at 08:19
  • apologies for the lack of formatting :-) – proteus Aug 11 '17 at 08:22

3 Answers3

4

When you add the message to the queue just set the initialVisibilityDelay so that the message isn't seen until the minimum process time.

CloudQueue queue = queueClient.GetQueueReference(queueName);
var msg = new CloudQueueMessage("Hello World!");
TimeSpan timeSpanDelay = GetEarliestProcessTime();
await queue.AddMessageAsync(msg, null, timeSpanDelay, null, null);

CloudQueue.AddMessage

jimmy_b
  • 163
  • 1
  • 7
3

So there're a few things with Azure Storage Queues that is going to help you with this scenario:

Regarding putting the message back in the queue, you don't have to do anything special. It is a feature offered by Storage Queues. When you dequeue a message (GET Message in Azure Storage lingo), the message becomes invisible for certain amount of time and if not deleted by the process which dequeued it will become visible again and can be picked up by another process.

So when you dequeue the message, check the time and if it is not the right time you don't do anything. However please make sure that once the message is processed, you delete that message otherwise it will be picked up again.

One more thing you could do is when you dequeue the message and find that it is not the right time to process that message, you update that message and set its visibility timeout property to a value that will make the message visible again closer to the processing time. For example, you dequeued the message at 18:00 and found that this message needs to processed at 19:00. In this case you will update the message and set its visibility timeout to 50 minutes (or a value greater than 30 minutes as 30 minutes is the schedule for your web job). What this will do is ensure that when your webjob runs at 18:30, this message will not be picked up by the web job because the message will only become visible at 18:50.

You can read more about updating message here: https://learn.microsoft.com/en-us/rest/api/storageservices/update-message and about dequeuing messages here: https://learn.microsoft.com/en-us/rest/api/storageservices/get-messages.

Update

I completely forgot that it is in a WebJob so doing nothing would actually delete the message. I guess you have 2 options (kind of repeating what's mentioned in the comments):

  1. Throw an exception instead of doing nothing. This will ensure that WebJob processor does not delete the message. I have not tried it myself but you can also update the message and set its visibility timeout to a value closer to desired time in the WebJob itself (and then throw an exception). This is kind of an anti-pattern though.
  2. You add a new message in the queue and set its initial visibility timeout to a value closer to desired time (this is also covered in another answer) and let this message delete.
David Peden
  • 17,596
  • 6
  • 52
  • 72
Gaurav Mantri
  • 128,066
  • 12
  • 206
  • 241
  • I havent actually used GetMessage to read the queue message, In my WebJob, I have host.RunAndBlock, then in my function class I have a method (ProcessQueueMessage) thats triggered whenever a new message is written tot thew queue, this was the 'default' webjob behaviour. Is this not best practice ? – proteus Aug 11 '17 at 08:39
  • the reading and deleting (and putting in the poison queue) all seems to be happeing automatically without any intervention from me, I just process the message when its detected – proteus Aug 11 '17 at 08:41
  • @proteus: You are correct, under WebJobs messages are deleted automatically when a function completes normally. Gaurav is stating the reverse, maybe he is familiar with a lower level Queue Storage API. – camelCase Aug 11 '17 at 08:45
  • so in my scenario where this reading and deleting is an automatic process, how would I 'ReQueue' ? – proteus Aug 11 '17 at 08:48
  • @proteus: Throwing an exception will cause an item to be returned to a queue but your situation seems more complex. – camelCase Aug 11 '17 at 08:51
  • I guess if doing that would put it back in the queue it would work, but it seems a very messy way of doing it, surely there must be a way to override the behaviour. Ive updated my original question to include my code that detects the queue message – proteus Aug 11 '17 at 08:57
  • @camelCase You're correct. I completely forgot about WebJobs part :). – Gaurav Mantri Aug 11 '17 at 09:00
  • @Gaurav: No problem, perhaps you could edit your answer to indicate that in some situations working outside of WebJobs allows more control of queue message handling. – camelCase Aug 11 '17 at 09:07
  • @camelCase I have updated my answer. Please feel free to edit it if you think it can be improved. – Gaurav Mantri Aug 11 '17 at 09:09
  • Im not really a fan of throwing exeptions in this way, so I think Ill go back to the webjob that places messages in the queue and set its visibility closer to the allotted time that should work – proteus Aug 11 '17 at 09:13
  • `so I think Ill go back to the webjob that places messages in the queue and set its visibility closer to the allotted time that should work` - I think its a very nice approach. – Gaurav Mantri Aug 11 '17 at 09:17
  • I've implemented that solution now and it works very well, thanks for your help on this problem guys, its very much appreciated, I've got a much better understanding of this process now., – proteus Aug 11 '17 at 11:05
  • @GauravMantri In your original answer, regarding Azure Storage Queues. In the doc, https://learn.microsoft.com/en-us/rest/api/storageservices/get-messages, "after the visibilitytimeout interval expires, the message again becomes visible to other consumers." - Does that mean the message is enqueue to queue again? – nullify0844 Sep 09 '19 at 16:15
  • @RyanEfendy - Yes, the message will appear in the queue again once the visibility timeout period expires. – Gaurav Mantri Sep 09 '19 at 16:32
1

When you en-queue an item in Azure Storage queues you can add extra detail to the item that will cause the item to be hidden for a configurable duration. If your batch job can only run every two hours but you want to delay sending emails with more fine-grained time control then I suggest the two-hourly en-queuing batch job could use this "initialVisibilityDelay" feature.

Here is another SO question that describes the API.

Azure storage queue message (show at specific time)

camelCase
  • 1,549
  • 2
  • 16
  • 28