Work Queue With Failed Job Reporting

Question

I have a WCF service (the fact that it's WCF shouldn't matter) and I'm not looking for message queuing, but instead for an asynchronous work queue in which to place tasks, once a request / message is received. Requirements:

• Must support persistent store that enables recovery of tasks in the case of Server / service process failure.
• Supports re-running of failed jobs, up to a given limit (i.e. try re-running a job up to 5 times)
• Able to record the failed job call along with its parameters, in an easily queried fashion. For example, I would query the store for failed jobs and receive a list of "job name, parameters".
• Unfortunately cannot be a cloud-based / hosted solution.

Queues that I'm probably not looking for:

• MSMQ (RabbitMQ, AMQP). Low level, and is focused on message transport.
• Quartz.NET. Has some of the above but its error-recording facilities are lacking. Geared more toward cron-like scheduling than async work and error reporting.
• the Default Task Scheduler of .NET TPL. It has no persistence of the process owning it stops abruptly and doesn't support re-running of tasks very well.

I think I'd be looking for something more along the lines of Celery, Resque, or even qless. I know Resque.NET exists (https://www.nuget.org/packages/Resque/), but not sure if there's something more mainstream, or if that could suffice.

Answer 1

What about Amazon SQS? You don't have to worry about infrastructure as you would with RabbitMQ/MSMQ. SQS is dirt cheap, too. Last time I checked, it was $0.01 per 10,000 messages. Why re-invent the wheel? Let Amazon (or other cloud providers with similar services, like Microsoft and Rackspace) do all the worrying.

I use Amazon SQS in production for all message-based services. Some of these messages act like chron jobs; an external process queues the message at a specific time. Some of them are acted upon immediately.