15

In Amazon Web Services, their queues allow you to post messages with a visibility delay up to 15 minutes. What if I don't want messages visible for 6 months?

I'm trying to come up with an elegant solution to the poll/push problem. I can write code to poll the SQS (or a database) every few seconds, check for messages that are ready to be visible, then move them to a "visible queue", or something like that. I wish there was a simpler, more reliable method to have messages become visible in queues far into the future without me having to worry about my polling application working perfectly all the time.

I'm not married to AWS, SQS or any of that, but I'd prefer to find a cloud-friendly solution that is stable, reliable and will trigger an event far into the future without me having to worry about checking on its status every day.

Any thoughts or alternate trees for me to explore barking up are welcome.

Thanks!

Hairgami_Master
  • 5,429
  • 10
  • 45
  • 66

3 Answers3

8

It sounds like you might be misunderstanding the visibility delay. Its purpose is to make sure that the polling application doesn't pull the same item off the queue more than once.

In other words, when the item is pulled off the queue it becomes invisible for a predetermined period of time (default is 30 seconds, max is 15 minutes) in case the polling system has a cluster of machines reading from the queue all at once.

Here's the relevant documentation:

http://docs.amazonwebservices.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/IntroductionArticle.html#AboutVT

...and the sentence in particular that relates to my comment is:

"Immediately after the component receives the message, the message is still in the queue. However, you don't want other components in the system receiving and processing the message again. Therefore, Amazon SQS blocks them with a visibility timeout, which is a period of time during which Amazon SQS prevents other consuming components from receiving and processing that message."

You should be able to use SQS for your purpose since you can leave an item in the queue for as long as you want.

Jay Gibb
  • 483
  • 2
  • 12
  • 2
    Hey Jay- Sorry I wasn't more clear with my question. I'm not referring to message visibility, but Delay Queues, where message are not visible to anyone for a period of time (and not because they've been checked out by some other process). This prevents messages from being processed at all by anything for a user determined period of time. Here's a link: http://docs.amazonwebservices.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/IntroductionArticle.html#sqs-delay-queues – Hairgami_Master Dec 20 '11 at 16:02
  • 2
    You may want to take look at http://www.iron.io, they've got a couple new products that look pretty good for what you're trying to do. – Jay Gibb Jan 03 '12 at 07:38
7

7 years later, and Amazon still doesn't support the feature you need!

The two ways you can sort of get it to work are:

  • have messages contain a delivery target datetime in their message_attributes, and have the workers that consume the queue's messages just delete and recreate any message that is consumed before its target, with delay = max(0, min(secs_until_target_datetime, 900)) ; that would allow you to effectively schedule a message for any arbitrary future time;

or,

  • (slightly less frequent and costly:) similarly, if a message isn't due to be handled yet, recreate it and change its visibility timeout to be timeout = max(0, min(secs_until_target_datetime, 43200))

The disadvantage of using visibility timeout is that any read will re-trigger it.

yivi
  • 42,438
  • 18
  • 116
  • 138
Ben Wheeler
  • 6,788
  • 2
  • 45
  • 55
4

There has been a direct AWS solution possible since 2016-12-01: AWS Step Functions

Each execution can last/idle up to one year, persists the state between transitions, and doesn't cost you any money while it waits.

Sean Summers
  • 2,514
  • 19
  • 26
  • 1
    Here it says the duration of the execution also is counted. https://aws.amazon.com/step-functions/pricing/ – adiga Sep 11 '22 at 13:55
  • 1
    Duration is calculated "per GB-Second". There is no CPU/memory being used during state transitions or waiting state nodes (there is, however, a charge for state transitions, so it's technically not "free"). This is my experience as person that pays an AWS bill for Step Functions already. AWS has many managed services that are completely free until use (AWS Lambda itself is free for storage and setup until (and only during) execution, which is the component Step Functions is composed from). – Sean Summers Sep 15 '22 at 14:03
  • Thanks for clarifying that. I assumed the GB-Second might not be high but since I'm keeping it in Wait state for a month, I thought it might rack up some cost just because the seconds will be too high even if it the computation part is very very small. – adiga Sep 16 '22 at 06:10