how safe it is to use Resque for long-time-pending tasks in Rails (for example: Resque.enqueue_in(7.days, JobClass)) with resque and resque-scheduler gems? Good to mention that instance is running on Heroku. What is the chance of losing queue?
Asked
Active
Viewed 851 times
1
xamenrax
- 1,724
- 3
- 27
- 47
-
If you're prepared to save your data in a DB, surely it's a similar principle with Redis. If you need to save them any longer (and cannot afford the possibility of the queue being reset), why not put them into some sort of persistent storage, like MYSQL, and then put that day's queue into Redis? – Richard Peck Oct 06 '15 at 09:07
-
@RichPeck, yes, I was thinking of storing such jobs in some persistent storage, but how could I save resque queue into DB? – xamenrax Oct 06 '15 at 09:11
-
Make a table for it with a model, and save any new jobs to it. Then each day, with a cron job (the "Heroku Scheduler"), you'll be able to pull *that day's* jobs into redis. Reseque will then be able to work through the Redis queue – Richard Peck Oct 06 '15 at 09:13
-
@RichPeck, ok, I got it, thanks, but is the chance of losing/resetting redis queue so big, that saving jobs to db will worth it? – xamenrax Oct 06 '15 at 09:24
-
Depends on how many items you're handling and how important it is. If you were, for example, using it for a Hotel reservations system (where welcome & information emails are sent to guests at specific times), I'd class that as vitally important. I'd save the data to the db (gives historical data), and then load redis up as mentioned. – Richard Peck Oct 06 '15 at 09:30
-
If, on the other hand, you were using emails in the capacity of special offers or something, and weren't that important, you'd be able to rely on redis. Either way, I think I'd still prefer to know which messages had been sent, so I could maintain the data to support it. – Richard Peck Oct 06 '15 at 09:32
-
1@RichPeck, thx for reply, you could rewrite these comments as answer and I will accept it. – xamenrax Oct 06 '15 at 09:34
1 Answers
3
Semi Persistence
As per my comments, the downside of using Redis (which Resque uses) is the potentiality of downtime.
As the "RAM" of web-apps, Redis stores semi persistent data. This basically means it only retains your data whilst it is operating. Any downtime would lose your queue.
This is the crux of your question -
can you afford to lose your queue?
The answer is only one which you can answer.
There are, however, a number of factors which I would consider:
*Redis isn't a full database... it's a JSON
key:valuepair store.
In short, this should only be used to give you a simple mechanism to use snippets of data at particular times. We use Redis to store ID's of users on one of our systems, as an example.
This means that if you're storing more data inside Redis than simple ID's or other base data, are you using it correctly?
How Important Is Your Queue?
As with any app, you'll want to retain as much user data as possible.
The difference, however, is how critical the queue is to your system.
I gave the example of a Hotel Reservation system. Perhaps you have "welcome" emails you need to send, "directions" emails to help guests find the hotel, etc. Are these "mission critical"?
I have designed an email marketing system before. The queuing system for that was extensive; but it wasn't the "queue" that was important - it was all the associative data which went with it.
Thus, we saved our data in a datatable and only used Redis to store a pure queue of what was going to be sent, and when. We then ran some scripts every minute (I've forgotten the specifics), which went through the Redis queue.
Bottom line is that I would highly recommend storing the appropriate data in a table, and then only processing the requests as you need them.
The underpin of the queue is that you need to be able to rebuild it if it goes down. If you're storing your values in redis alone, this will prove to be a major bottleneck.
Hitting the DB is well worth the data you'll get from it in the long run.
Richard Peck
- 76,116
- 9
- 93
- 147