General-purpose distributed scheduling library for Java

Question

I'm looking for some general library for scheduling lots of tasks. The library has to provide ability to split tasks across nodes in cluster, perform load balancing and fault tolerance - so if some node goes down, the tasks for the node has to be distributed across remaining nodes.

I looked at the Hadoop - but look like it will work well for map-reduce tasks. In my case tasks are simply senders of notifications, checkers for object state etc.

Quartz seems to be great - but it's not clear how good is it when it comes down to dispatching events to nodes.

Any other options?

score 7 · Accepted Answer · answered May 16 '12 at 13:09

7

Sounds like a use case for Akka.

answered May 16 '12 at 13:09

Christian Schlichtherle

3,125
1
23
47

1

or for [Redisson scheduler service](https://github.com/mrniko/redisson/wiki/9.-distributed-services/#94-distributed-scheduled-executor-service) – Nikita Koksharov Aug 25 '16 at 14:40

score 3 · Answer 2 · answered May 16 '12 at 13:26

3

I agree with Christian that Akka seems like a better fit. Quartz is great for what it does, but its basic building block is a Job that should be executed. It's not going to help you decompose your job into distributable components.

If all your tasks can be decomposed into jobs, then Quartz can help you schedule them, that's what it does best. But if a job needs to decompose into subtasks, then you'll need to use another framework.

Another option might be spring batch, depending on your needs.

answered May 16 '12 at 13:26

Paul Sanwald

10,899
6
44
59

For now, it is possible to run one task as "job", but I don't see how quartz can deal with distributed task management and handle cases when one node is down. And it uses polling , so it is needed to configure some sort of database where job has to be stored, this is potential bottleneck. Am I missing something? – jdevelop May 16 '12 at 14:22
the database is indeed a bottleneck. for distributed tasks, you can configure clustering, but this is somewhat limited – Paul Sanwald May 16 '12 at 18:45

Youri · Answer 3 · 2014-09-10T09:06:05.847

1

You could try ProActive, a workflow scheduler that integrates well with Java (written in Java). It supports the features you listed. It is an open source project, part of OW2.

The following features for instance are available:

Workflows, ie set of tasks with dependencies and constructs such as loop/if/replication
Tasks are executed on nodes, i.e agents running on different machines
Tasks can be restarted automatically upon node failure

[disclaimer] I'm part of Activeeon, the company providing professional support for ProActive[/disclaimer]

edited Sep 10 '14 at 09:06

answered Sep 08 '14 at 16:05

Youri

51
3

Please read [answer], answer should also include enough information to be useful even if the link dies. – Jonathan Drapeau Sep 08 '14 at 16:36

score 0 · Answer 4 · answered May 16 '12 at 13:15

0

Or you can also try Quartz, it can handle almost anything you want:)

answered May 16 '12 at 13:15

Petr Mensik

26,874
17
90
115

what about distributed task processing and cases when one node goes down? Looks like there is no out of the box solution. – jdevelop May 16 '12 at 14:23
@jdevelop Quatz clustering mode does support fail over – Duc Tran Oct 23 '19 at 17:59

General-purpose distributed scheduling library for Java

4 Answers4