0

I am trying to implement a cluster computing system using Java and the Windows OS. I'm looking for a solution that

  1. is not too out of date
  2. is reasonably easy to install and set up
  3. has enough documentation to get started with the classes and methods without a previous knowledge of MPI
  4. is at least somewhat user friendly

This might not be possible, but it would also be nice if it was somewhat close in usage to the Java Concurrent framework.

I initially learned a bit about the Java Concurrent package and was easily able to learn to write parallel programs on my local 8-core machine using the Runnable Interface and the ExecutorService, making all my classes threadsafe in the process. However, I have yet to find a standard mechanism to extend this programming framework to clusters.

I then learned of a GitHub project called Java-Interop-Library (https://github.com/MicrosoftHPC/Java-Interop-Library ) that could be used with Microsoft HPC Pack. I networked a few cloud computer via Amazon EC2 and installed the HPC Pack. The Java-Interop-Library was a nightmare to compile and set up. I had to manually edit several batch files and even some Java code to get it compiled. By the time I got most of it working (but not everything), I just started thinking that there had to be an easier way, and I started searching again.

My new search led me to MPJ-Express (http://mpj-express.org). I read through the documentation on the site, and it seems easy to set up. They even have documentation on how to integrate it with Eclipse and debug. But, I could never find any documentation on how the classes and methods are actually used (there's a simple hello world example, but it's not close to enough).

More searching led me to MPIJava, Hadoop, and GridGain. Having no experience with MPI or MPJ, and knowing that MPJ grew out of MPIJava, I started trying to find documentation for that instead. I found some docs, but some of it is quite old and I'm not really sure I'm on the right track. I saw the mention of GridGain on another StackOverflow post, and went to their website. They seem to have a cluster computing framework, and a simple posted example even uses what appear to be classes that use Runnable objects, which seemed attractive to me having had some experience with the Java Concurrent framework. I don't really know anything about Hadoop, other than that it might be a possibility.

I really just need some better direction on the best way to accomplish cluster computing i Java. I feel like I'm just spinning my wheels.

Bill the Lizard
  • 398,270
  • 210
  • 566
  • 880
RyanS
  • 77
  • 1
  • 9
  • 3
    Questions asking for tools or library recommendations tend to be closed. However, although it's not entirely clear what you actually want to achieve in the end, you might want to have a look at http://hazelcast.org/ . The installation+setup is rather simple: **Just use the JAR**. (That's it, really - at least for the first tests. For 1000 nodes, it might be more involved). And the best thing is that it offers a *distributed* version of the infrastructure that every Java developer knows: Particularly, it offers a distributed `ExecutorService`. – Marco13 Jun 12 '14 at 20:15
  • @Marco13 As you know this you, and the person who up voted you comment, should be voting to close this question as off topic. – Raedwald Jun 13 '14 at 08:25
  • @Raedwald Maybe the upvote was for the "answer-ish" part of the comment? I see the problem with this kind of questions: They tend to be subjective, have a potential for flamewars, and in the end, the only *right* answer is always the same: "That depends on what *exactly* you want to do". But an answer containing a good and objective comparison of different alternatives can be very valuable, so I leave it to others to vote for closing, if they want to. – Marco13 Jun 13 '14 at 08:39
  • I'm not positive about this, and the purpose of this question is to try and clear this up, but there seems to be a lack of any consensus regarding what framework to use for cluster computing using Java. Without any standard framework, everyone is going to learn something different, making it more difficult to share ideas, code, etc. There might be some benefit if the Java community discusses this so that a proper direction emerges. If this is not it, does anyone know of a proper forum for this discussion? – RyanS Jun 13 '14 at 14:18
  • @Marco13 hazelcast seems awesome. Thanks again for the suggestion. It really seems to extend the existing Java concurrent package to clusters. There also seems to be good introductory documentation and even a book. – RyanS Jun 14 '14 at 16:23

1 Answers1

3

You can use MPJ Express and I assume that you have already given it a go and environment set up is done. MPJ Express can execute parallel Java applications on both multiocore mode and cluster mode. It must be mentioned there that the same application works for both modes without any modification so you have the option of developing and testing your application in multicore mode and when done it would seamly run cluster mode as well. Although It is not quite clear what you want to develop in the end but as far as its Java docs are concerned you can find them here: http://mpj-express.org/docs/javadocs/index.html. These are all the classes and methods that you need to develop any parallel java application.

Learning cluster Programming with MPJ Express is quite easy as you can access a comprehensive set of working examples. For this purpose unzip MPJ Express and explore test directory. It contains multiple test cases and working examples of parallel Java applications. You can have examples of point-to-point and collective communication in mpj-v0_xx/test/mpi/pt2pt and mpj-v0_xx/test/mpi/ccl respectively. You can develop any basic parallel application by just using point-to-point and collective communication methods. It would give you head start in learning cluster Programming using Java.

Let me know if you encounter any problem in setting up MPJ Express on cluster or executing examples provided in test directory. You can also post your queries on MPJ Express mailing list on following page: http://sourceforge.net/p/mpjexpress/mailman/?source=navbar

Ansar
  • 131
  • 3
  • Thanks for the response. I'm curious if there is any introductory documentation about using MPJ Express for someone not really familiar with MPI. Javadocs and code examples are always nice, but I need a more basic introduction written in English to start me out. I think I've gotten the basic idea from the code examples... you find your rank and then use either a Send or Receive method depending on whether you're a manager or worker. However, I can't reconcile this with what I've learned through the Java Concurrent framework. i.e., making classes threadsafe, sending Runnables, etc. .... – RyanS Jun 13 '14 at 15:25
  • ...For example, will the MPJ Express framework understand and respect what the keyword "synchronized" means? So far, it seems like you just send low level data to worker nodes via send and receive. But on the other hand, the if/else structure code using send/receive must also go off to the worker nodes somehow for them to know what to do with the sent data, but somehow this does not go through the send/receive mechanism? If all the code is going to the worker nodes anyway, and if all nodes rely on a low-level if structure to determine what they should do, why do you even need send/receive? – RyanS Jun 13 '14 at 15:25
  • Ok, so I can imagine a scenario where the code (or bytecode) is distributed to every node for computing purposes, but the memory that the code references is not sent by default, in which case you need to send it via a Send method. I'm probably totally off track here, but at least it doesn't seem totally non-sensical to me anymore. I still need a basic, English-written introduction if it exists. – RyanS Jun 14 '14 at 16:21
  • 2
    @RyanS Yes the MPJ Express works on the principle of Distributed Memory. When an MPJ Express program is launched on multiple nodes what happens is that multiple instances of that programs are initiated on those nodes. Each instance which runs the same code is actually identified by a unique identifier called the `rank`. I suggest reading basic material on whats MPI and how to program in MPI. A comprehensive guide is underway but don't know when it will be released. – Bibrak Jun 17 '14 at 09:14
  • @RyanS Also it will be helpful if you could share what kind of problem you are trying to solve using MPJ Express or "Parallel Java Frameworks.." for that matter. – Bibrak Jun 17 '14 at 09:17
  • 1
    @RyanS there are a lot of tutorials and lectures available for parallel programming. You need to just use search engine for that I am sharing one with basic details [link](https://computing.llnl.gov/tutorials/mpi/) that may be suitable for you but you should not limit yourself to it. For development in parallel computing you should know about Single Program Multiple Data (SPMD) model and why message passing is required. It would be good if you can share basic idea of what you want to develop else this discussion would go endless. – Ansar Jun 17 '14 at 10:35