4

Azure Message Queue messages have a maximum time to live of 7 days. Ugh why? I'd like my message to have an infinite time to live and wait in the queue until I get around to processing it. If not for this bizarre 7 day limit Azure queues would be a perfect solution for me. I have a whole 100TB storage account backing my Queue, why can't I use it?

I'm hoping someone has an idea for a workaround or solution to this problem. Any ideas?

BowserKingKoopa
  • 2,701
  • 3
  • 33
  • 40

4 Answers4

3

You can now opt-in to infinite TTL by specifying an expiry of -1 seconds when you queue the message. This is new with api-version 2017-07-29:

For the Queue service, the Put Message API now allows a time-to-live value in the messagettl parameter of over seven days. You may also specify -1 for this parameter to indicate that the message should remain in the queue until dequeued and deleted. The default value for this parameter is still seven days.

If you're stuck with an earlier version of the Azure Storage libraries, I solved this problem by requeuing old messages - once they were 6 days old, I added a duplicate to the queue and deleted the original.

Bevan
  • 43,618
  • 10
  • 81
  • 133
  • Awesome! I just tried it out and it seems to work. When I pass in a value of -1 seconds Azure sets the Queue message expiration date to 12/31/9999, 6:59:59 PM. So I guess I don't have to worry about this for a while. – BowserKingKoopa Apr 02 '18 at 23:14
2

Service Bus Queues give you unlimited TTL as Brian points out.

You can get over the 5GB limit by sharding across multiple SB Queues. What we find, though, is that people who hit that limit will commonly find and approach more useful and cost effective whereby whatever large data items they have go into Storage (i.e. Blobs) and only the jobs associated with those go into the Queue.

That will allow you to leverage storage's ability to do block-wise uploads (with block blobs) and then communicate the presence of that blob through an SB queue. Once you've processed the job you can then remove the data.

There aren't a lot of common use-cases where a Queue will exceed the 5GB cap and that sort of pattern isn't the overall better choice.

Clemens Vasters
  • 2,666
  • 16
  • 28
1

You might look at Service Bus Queues. I'm not super familiar with them but in some cases they are more flexible than Storage Queues. You could also look at other queueing systems (MSMQ, RabbitMQ, etc) but they are probably not worth the effort to set up in Azure.

Another option is to use a Storage Table as a Queue. You can then support any TTL you like. There's some discussion about doing that here. I did actually implement that system and it works fine. If I get a chance I'll try to post the code.

Brian Reischl
  • 7,216
  • 2
  • 35
  • 46
  • If I'm not mistaken Service Bus Queues have a maximum size of 5GB for some reason which makes them less than useful for my application as well. I'm currently using Azure Tables but they're a poor replacement for the Queues and I find myself writing awkward code to make up for it. – BowserKingKoopa Jul 01 '13 at 00:01
  • 1
    That is a lot of data - could you maybe store the data in tables/blobs and just enqueue a pointer to the data instead? I didn't have a lot of problems with the coding though, it's not terribly hard to make it expose the same API as the regular queue service. – Brian Reischl Jul 01 '13 at 01:38
  • If you're using Service Bus Queues, there is an option on the Queue object called EnableDeadLettering. This will put any expired messages in a dead letter queue where it can be managed as such. – ScottB Oct 26 '15 at 04:26
1

I think the 7 day limit you are referring to is the visibility delay (i.e. time that the message will be invisible)

For time-to-live (specifies how long a message will remain in the queue) there is no documented limit that I aware of ( please point me to a link if this is not the case)

I am basing my comment on the following page http://msdn.microsoft.com/en-us/library/windowsazure/hh563575.aspx

Hope this helps

Vishwas Lele
  • 46
  • 1
  • 1
    I believe this is wrong, though is does not say what the default ttl is there is a one of 7 days. If you look at the rest api (http://msdn.microsoft.com/en-us/library/dd179346.aspx) it says there is a default of 7 days. – lockwobr Apr 15 '14 at 18:03