4

I have a producer that reads from a twitter stream and places tasks on a queue using rabbitmq as the message broker.

The jobs usually takes around 3-5 seconds to execute and at the end of the job I would notify the user when it is done. However, I would also like to remind them in 10 minutes/30 minutes/60 minutes after their job is done by sending the user more messages.

I cannot figure out the best way to schedule these additional messages. I thought of creating cron jobs to run a message script at those specific timestamps, but that seemed wrong with thousands of jobs an hour. Another idea was to run a daemon that would poll a database or read from queue and checks the scheduled timestamp and current time before sending out.

Ideally if this could be done on the message broker side through delayed messages it would make it much cleaner, but I read that RabbitMQ does not support something like that.

What would be the best way to go about scheduling many jobs that needs to be executed by timestamp?

James Cowhen
  • 2,837
  • 3
  • 24
  • 32

1 Answers1

0

Just create a single cron job that runs every minute and checks another queue, which has timestamps for when the message will be sent. When the timestamp has passed, send the message and remove the item from the queue.

DaveRandom
  • 87,921
  • 11
  • 154
  • 174
  • If I wanted more granularity, would running cron job every few seconds to check be excessive? Is there a better way than that? – James Cowhen Jan 12 '12 at 19:39
  • Well, first of all, you can't run cron jobs more than once a minute unless you stagger them by sleeping. You don't have to schedule a cron job for each message - what about a script which sends out 50 messages at once? – andrewtweber Jan 12 '12 at 19:50