3

I have a lot of trivially parallelizable computations and a lot (100s) of cores distributed overs SSH + NFS network. What is the simplest way of parallelization. The problem is that I don't know how long each task will take so I need some kind of queue. Is there something that is very easy to use?

Łukasz Lew
  • 48,526
  • 41
  • 139
  • 208

3 Answers3

1

Mincemeat.py is a lightweight Python implementation of MapReduce designed just for this purpose. There's not an easy way to launch all of the processes (unless you have something like Condor on the cluster), but otherwise it should fit your needs.

Michael Fairley
  • 12,980
  • 4
  • 26
  • 23
1

You may also have a look at mpssh & autossh.

http://www.mynitor.com/2010/08/16/top-50-ssh-helper-tools-omg/

telld
  • 11
  • 1
1

I have a lot of trivially parallelizable computations and a lot (100s) of cores distributed overs SSH + NFS network. What is the simplest way of parallelization. The problem is that I don't know how long each task will take so I need some kind of queue. Is there something that is very easy to use?

I'm answering late, but I went on a similar search. I've been using this for a while and I'm pretty convinced that orgalorg is the answer.

Linux and macOS are supported.

Example below...

Mike Pennington
  • 41,899
  • 19
  • 136
  • 174