6

I'm looking for a tool for running a series of commands like the existing tool:

parallel -h
parallel [OPTIONS] command -- arguments
    for each argument, run command with argument, in parallel
parallel [OPTIONS] -- commands
    run specified commands in parallel

But I'd like these commands to be run over ssh on multiple computers, with some of the niceties of pssh or pdsh for communicating with many hosts. I've hacked out something that works,, but it's ssh handling is nothing compared to these tools - I can't stop them all, or even see all of their outputs.

Even better if the tool has some basic load balancing, but I was thinking I'd use a separate tool for host selection. (A good tool for querying load, memory, and if a computer is in interactive use would also be appreciated, but I've already written something that will suffice for host selection.) This isn't on a cluster, and I don't want to rely on daemons other than sshd, or ask admins to install a serious cluster job scheduler like Condor. I don't have root access on any of these computers.

Edit: To emphasize, I want to run different commands on each host - typically running the same program with different arguments, as in the first parallel usage example above.

Community
  • 1
Thomas
  • 191
  • 1
  • 9
  • related question: http://serverfault.com/questions/13322/what-distributed-shell-utilities-do-people-feel-are-good-flexible-and-easy-to-u But I checked out every one of these, and none of them seem to allow distinct commands to each host. – Thomas Apr 13 '11 at 14:31
  • Ah! It looks like the GNU version of `parallel` does do this - http://www.gnu.org/software/parallel/man.html#example__using_remote_computers. No load balancing, and I haven't tried it out to see what it does with each stdout, but I think this is precisely what I wanted. Unfortunately I've written a script that incorporates some load balancing, so I'll be sticking with it for now. – Thomas Apr 25 '11 at 21:44

5 Answers5

3

Ah! It looks like the GNU version of parallel (not the one I had installed) does do this. No load balancing, and I haven't tried it out to see what it does with each stdout and stderr, but this is precisely what I wanted. 

To run commands on more than one remote computer run:
seq 10 | parallel --sshlogin server.example.com,server2.example.net echo

Unfortunately I've written a script that gives status updates, has configurable output settings, and incorporates some simple load balancing, so I'll be sticking with it for now.

Thomas
  • 191
  • 1
  • 9
1

Blockquote Edit: To emphasize, I want to run different commands on each host. Blockquote

if you want different commands where is the parallel part? parallel means to start the same command on an collection of hosts (running in parallel) ... if you want to do different things on different hosts that is an sequential process

Adrian Sevcenco
  • 166
  • 6
  • I mean parallel as in parallel processing, or parallel computing - the different commands are most likely the same command run on a different portion of the whole dataset. If I'm gziping 100 files, I want to divide these 'commands' (there're all the same command, gzip, with different arguments) among different hosts. (assume a shared filesystem for this example) – Thomas Apr 14 '11 at 15:19
  • Ok! thats different from what you have stated! you need to run the SAME command (process, function, etc) on different sets of arguments or through something that automatically split you arguments set and apply the "command" in parallel to the sub-sets.. – Adrian Sevcenco Apr 14 '11 at 19:03
  • and parralel is what you need .. only with a bit of scripting .. i imagine that you want to start on n hosts (that have m cores) a command with a set of arguments that it will be split in n X m pieces.. – Adrian Sevcenco Apr 14 '11 at 19:06
  • as i dont have so many machines i cannot make the script for you .. but i can give you same advices: 1. you will have to implement a merge function (for the results) that will be applied twice (on the processing host (partial results) and on the submitting host (final results) including the moving of partial results to submmiter) 2. argument splitting procedure must take into account at least the n X m number (this should be known) as partial results can come back at different times .. and you must sync and wait for all jobs for the final merger – Adrian Sevcenco Apr 14 '11 at 19:07
  • at this point if you have so many machines it could be better to use some job scheduler (lets say like torque or condor) (if you have a sum of 8 cores (2 machines) you could play with those .. i am more familiar with torque but for home use and ease of usage and installation documentation etc i would advise to play with condor http://www.cs.wisc.edu/condor/) – Adrian Sevcenco Apr 14 '11 at 19:14
  • What I'd like to do doesn't require a merge (or split) step; I'd like to use this for tasks so embarrassingly parallel that they're already conveniently split for me, as in the gzipping 100 files example. I'm somewhat familiar with Condor, but as far as I can tell I'd need root access (on all machines) to install it properly. – Thomas Apr 14 '11 at 22:08
  • yes, thats true ... than you are back to the script part. as for the merge, merging could mean only to copy all out files to a single director.. the "merging" part is just a concept. so, yes, in your case would be enough to copy back the gzipped files (btw .. are you aware of pbzip2?) – Adrian Sevcenco Apr 14 '11 at 22:20
0

You really should look into one of the many clustering technologies out there. Try looking at Apache Hadoop. I recently read a great article that you may find interesting on the subject too about setting up a 10,000-core cluster to do parallel computing: http://goo.gl/A8hgX

TheCompWiz
  • 7,409
  • 17
  • 23
  • I'd like to parallize existing scripts over ssh - not rewrite algorithms to use map-reduce or install large software on dedicated hardware. (I'm interested in those, but I need to leverage existing programs) – Thomas Apr 13 '11 at 14:39
  • 1
    Doesn't seem to be relevant to his question. And in fact, he even states in his question that it isn't a cluster and he isn't looking to have to install serious clustering tools. – Scrivener Apr 15 '11 at 17:37
  • @Scrivener... Actually what he is attempting to do is exactly what a standard computational cluster is supposed to do. Rather than trying to re-invent the wheel, I suggested a very common framework for doing the same thing that he was trying to do manually. And as "Thomas" mentioned in his own comment to me, he is interested in the technologies... just not at this moment. So, it would be more prudent to stop giving -1 to people simply because you don't understand the subject. – TheCompWiz Apr 15 '11 at 18:23
0

I've used mussh for this, it's bash based but runs in parallel. I'm pretty happy with it.

I've also seen a few talks for rshall (which despite holding RSH in the name, uses ssh natively) at the local Linuxfests, it's perl based and can use an external source for querying host lists, but it expects certain host information in specific formats.

Neither of these have queuing or job scheduling, although you could run them via cron or at if you wanted to.

None of these require root access but they do require you have key based auth to the systems.

dgibbons
  • 86
  • 1
  • As far as I can tell, both mussh and rshall just run the same command on each host - I'd like to run different commands on each host. – Thomas Apr 13 '11 at 15:57
0

clusterssh is another tool that might be worth looking into. It's more interactive in that it will open and tile terminal windows for each host. You can also run commands in each terminal separate from each other or in all (or some) at once. For example, running top on 12 systems at a time then chasing down a process in just one of them.

zilla
  • 89
  • 2
  • Again, it looks like `clusterssh` is just for running the same command on many hosts - I'd like to run different commands on each host. – Thomas Apr 14 '11 at 00:21
  • Not just, it's pretty flexible. However, it seems you're looking for a shell script that does something like "ssh host1 'command 1'; ssh host2 'command 2'"... – zilla Apr 14 '11 at 00:51
  • Yeah, that bash script is pretty much what I want, but with some of the nice features of things like `clusterssh`. I couldn't find a lot of documentation for `clusterssh`; I only found references to it controlling many of its concept of terminal sessions - what else does it do? I only meant "just" to mean that like the other two tools mentioned so far, it doesn't do quite what I want. – Thomas Apr 14 '11 at 03:27