0

I'm exploring azure functions and queue triggers to implement a recursive solver.

The desired behavior is as follows:

  1. There exists a queue called "solveme" which when it receives an item containing state, will create a second queue with a unique name (GUID) and send the initial state to that queue with some correlation information.

  2. When a message is received into the newly created queue, it will possibly enqueue more work representing all the next possible states so that before the message is "completed", the new work will exist in the queue. If there are no valid next states for the given state, it will just complete the message. This means that the original message is not "completed" until all new work is enqueued. During this step, I'm writing off data to azure tables representing the valid solutions using the correlation information from step 1.

3. When all messages have been processed from the newly created queue, delete the queue.

I am not sure how to go about #3 without having a timer or something that checks the queue length until it is zero and deletes it. I'm trying to only use Azure Functions and Azure Storage - nothing else.

It would be nice if there was someway to do directly from a QueueTrigger or within a function called by a QueueTrigger.

Further, is there a better / standard way to trigger completion of this kind of work? Effectively, the empty queue is the trigger that all work is done, but if there is a better way, I'd like to know about it!

Janusz Nowak
  • 2,595
  • 1
  • 17
  • 36
MPavlak
  • 2,133
  • 1
  • 23
  • 38

1 Answers1

3

There's no reliable way to do that out of the box. If your consumer is faster than producer, the queue could stay on 0 for long time, while producer still keeps sending data.

I would suggest you to send a special "End of Sequence" message, so when consumer receives this message it knows to go and delete the queue.

P.S. Have a look at Durable Functions - it looks like you could use some of the orchestration there instead of implementing workflows yourself (just mind it's in early preview).

Mikhail Shilkov
  • 34,128
  • 3
  • 68
  • 107
  • there should be no way for it to go to zero since the message is not completed until all the next work is added unless I'm missing something. Also, how can I send an end of work message in this recursive case? I do not see any possible way. – MPavlak Jul 13 '17 at 15:51
  • I will look into the Fan-In example from the link provided. Effectively I'm trying to do fan-in and am not sure how to do it using only these technologies. I'm find using durable functions if that accomplishes it. – MPavlak Jul 13 '17 at 15:53
  • @MPavlak I'm not sure what kind of "complete" you mean. Function will process messages one by one, but several messages of the same queue may be processed in parallel. I guess I don't understand your scenario good enough, maybe you could add an example? – Mikhail Shilkov Jul 13 '17 at 19:41
  • Will update when I get home, but messages are read as peek locked so that if the function crashes it can read again. Complete is signaling actual removal from queue. – MPavlak Jul 14 '17 at 00:21
  • @MPavlak what kind of approach did you use at the end to achieve this ? I have similar case but cannot use durable functions because they are not available for Python yet. – Rajat Arora May 18 '20 at 19:36