Moving long-delay queue from beanstalkd to AWS

Question

I am now moving job queue from local beanstalkd server to Amazon SQS. But the problem is that SQS can only have short delay (15 min.) for a job.

In my application, there are thousands of deals, and I use beanstalkd to expire the deals (i.e. call api to update deal status to completed/failed) when it meets the expire time.

Is there any service/hacky way I can used to handle this use case?

Thanks

Possible duplicate of [SQS - Delivery Delay of 30 minutes](https://stackoverflow.com/questions/31881470/sqs-delivery-delay-of-30-minutes) — vahdet, Feb 15 '19 at 12:21
What's your long-delay period? We've been fine with Beanstalkd and adding long jobs on Redis. Works amazing. — Pentium10, Feb 15 '19 at 21:12

score 1 · Answer 1 · answered Feb 16 '19 at 17:41

I'd just stick the data for a long-delayed-event in a database, and run a cronjob every ~10-15 minutes to suck in a group of upcoming jobs and write out to SQS, before deleting from the database. There will be some things to think about in terms of how many could arrive at once and trying to avoid reading the same jobs twice, and so inserting them again into SQS.

While databases aren't good for queues, that's mostly an issue with high interactivity and fast-turnover. I'd classify your issues more as things to come back to in several weeks or more.

You can even throw a regular message into the queues to go and fetch the next batch of upcoming jobs.

Moving long-delay queue from beanstalkd to AWS

1 Answers1