0

I have a project which briefly is as follows: Create an application that can accept tasks written in Java that perform some kind of computation and run the tasks on multiple machines* (the tasks are separate and have no dependency on one another).

*the machines could be running different OSs (mainly Windows and Ubuntu)

My question is, should I be using a distributed system like Apache Mesos for this?

The first thing I looked into was Java P2P libraries/ frameworks and the only one I could find was JXTA (https://jxta.kenai.com/) which has been abandoned by Oracle. Then I looked into Apache Mesos (http://mesos.apache.org/) which seems to me like a good fit, an underlying system that can run on multiple machines that allows it to share resources while processing tasks. I have spent a little while trying to get it running locally as an example however it seems slightly complicated and takes forever to get working.

If I should use Mesos, would I then have to develop a Framework for my project that takes all of my java tasks or are there existing solutions out there?

To test it on a small scale locally would you install it on your machine, set that to a master, create a VM, install it on that and make that a slave, somehow routing your slave to that master? The documentation and examples don't show how to exactly hook up a slave on the network to a master.

Thanks in advance, any help or suggestions would be appreciated.

Tom
  • 330
  • 1
  • 3
  • 17
  • 1
    Does you use case match traditional [grid computing](https://en.wikipedia.org/wiki/Grid_computing)? – marbu Sep 01 '15 at 22:59
  • @marbu I had a quick look at it but haven't considered any solutions yet. I will have a look into it tomorrow, a quick Google search has thrown up http://www.jppf.org/ which looks promising. – Tom Sep 01 '15 at 23:21
  • Maybe? All the major alternatives similar to Mesos seem to be mentioned with a little description here http://docs.deis.io/en/latest/customizing_deis/choosing-a-scheduler/ (no idea what the product of that website is though) but all those systems are complex to setup and don't solve your problem yet. That's all more for operating large scale deployments, not really specialized for computational tasks. – zapl Sep 01 '15 at 23:36
  • Take a look at Akka. You can build systems like you propose easily and with less overhead than using Mesos (nothing wrong with Mesos, it just may be overkill). – sourcedelica Sep 02 '15 at 17:30

1 Answers1

2

You can definitely use Mesos for the task that you have described. You do not need to develop a framework from scratch, instead you can use a scheduler like Marathon in case you have long running tasks, or Chronos for one-off or recurring tasks.

For a real-life setup you definitely would want to have more than one machine, but you might as well just run everything (Mesos master, Mesos slave, and the frameworks) off of a single machine if you're only interested in experimenting. Examples section of Mesos Getting Started Guide demonstrates how to do that.

hartem
  • 411
  • 2
  • 8
  • One addition: Windows support in Mesos is WIP. But why should it matter for JAVA tasks? – rukletsov Sep 02 '15 at 10:05
  • @rukletsov It shouldn't matter but I thought I'd point it out in the question just to be sure. – Tom Sep 03 '15 at 10:52
  • @hartem thanks for the information! I have had a look at the Mesos getting started guide and I'm in the process at the moment of trying to get Mesos running on two machines, one being the slave and one being the master which isn't actually explained that well in the documentation. – Tom Sep 03 '15 at 10:53
  • Actaully @rukletsov it does matter because as far as I know, you can't actually run Mesos on a standard Windows OS, it has to be Windows Server and the majority of the machines I am going to use will be running a standard Windows OS (e.g. Windows 7). – Tom Sep 03 '15 at 15:03
  • Well, then, unfortunately, you should probably wait or help us out changing it. Right now Mesos exists predominantly in the *nix world. – rukletsov Sep 03 '15 at 16:44