0

I need to implement a job queue for a distributed JVM application.

The queue(s) must hold a group of jobs (i.e. Java runnables) that have to be executed in sequential order. There will be hundreds of those "job groups" or queues that should automatically be parallelized on a cluster. The job qeues must be failsafe, so that in the event of a machine crash, the whole application will run on.

Also I do not want to re-invent the Weel, so if there is already a library or framework that can meet my requirements I would appreciate that.

Question: What technology should I use for it? Is akka appropriate, are there any other libraries you can recommend?

taffer
  • 407
  • 1
  • 5
  • 13

3 Answers3

2

In older days jGroups or ZooKeeper were the standard way to go. However I find that Akka is more flexible and takes on more legwork from you.

One of the features that ZooKeeper has over Akka is state replication, i.e. when some job does not make it into the queue or process crashes while executing the job, Akka does not recover it automatically. This however gives you a great throughput and also discards any illusions about the perfect world - which we know we do not live in. So you'd have to think about real world burn scenarios early on.

Mirko Adari
  • 5,083
  • 1
  • 15
  • 23
  • By the way, automatic replication is on the [Akka roadmap](https://docs.google.com/document/d/18W9-fKs55wiFNjXL9q50PYOnR7-nnsImzJqHOPPbM4E/edit?hl=en_US). – Rich Dougherty Jan 21 '13 at 23:38
0

Akka is definitely appropriate. Check out Akka's Durable Mailboxes to handle your "failsafe queue" requirement.

sourcedelica
  • 23,940
  • 7
  • 66
  • 74
0

Reliable channels of the eventsourced library may help you. Eventsourced adds persistence to stateful Akka actors together with at-least-once message delivery via channels and allows you to recover actor and channel state from JVM crashes. This example shows how you can use the library with akka-cluster. The example uses transient channels but you can replace it with reliable channels by using ReliableChannelProps at this line in the example. When a reliable channel fails on one node it will be automatically recovered on another node in the cluster.

Martin Krasser
  • 745
  • 5
  • 13