-2

Consider the following:

I (hypothetically) have one heavyweight server/master node with a couple disks in it and a group of lightweight diskless clients/nodes on a 10 gigabit LAN.

The server is running: DHCP and TFTP to give the clients a kernel over PXE. PXE kernel loads and mounts NFS on master server as root filesystem.

I route all internet access through the master server and share it with the clients to protect from possible security problems introduced by the fact that I want to maximize I/O between the nodes and the master, thus creating a private unprotected trusted network and a public firewalled network. This would be made possible by the fact that my master node would have 2 gigE ports.

The goal is to set up a software development lab for a group of about six people for less than four thousand dollars. The idea is to share as much of the hardware as possible.

My question is this: Which is better, to do compiles directly on the NFS mount, or do compiles on a tmpfs (a fileystem completely in RAM with no backing store) and then have my swap on the master node (and hope I don't have to use it too much). Or is it worth it to just get SSDs for the slave nodes so they can do local compiles and then save the results to their NFS mounts on the master, keeping in mind that this is very expensive and contrary to the goals of the setup.

Max DeLiso
  • 103
  • 3
  • Two comments: you might want to consider migrating to the SO sister site, http://superuser.com. Second, have you considered a VMWare setup with a thin client? You could create virtual systems quickly and set up a virtual SAN to share data. –  Sep 15 '11 at 20:05

1 Answers1

1

I've set up multiple systems like this. It's also handy to have a boot disk/USB that can hit the DHCP and TFTP sites to load an image to any random system that you want to temporarily add to your cluster. You can also set up workstations to automatically reboot into your cluster on a nightly basis (for some off-peak processing power), and reboot into whatever other OS during the day. By the way, if you're doing this by hand, there are lots of toolkits out there to help.

To answer your question, the general way to do compiles on a system like this is to compile on the frontend with all of the disks, and have the home directories NFS mounted from there to the compute nodes. If you put it on the tmpfs, you'd have to rebuild a node any time you recompiled. If you ever scale out to hundreds or thousands of nodes, NFS might not scale that high, but tools like autofs can help with that and nfs will work fine for your needs. I assume you're setting this system up to run some sort of parallel jobs (MPI, hadoop, etc.). I mention this because you probably want the 6 users to share this parallel resource in an effective manner. I'd suggest using a queuing system for jobs so that a single compute node isn't shared by more than one user at a time (and a single job could span many/all of your compute nodes).

SSDs might be nice, but I think they completely defeat your goal of having an inexpensive parallel compute resource. As long as you have enough RAM to hold a small image and run your jobs, NFS will get the application code to them just fine. If you can't fit your job in memory, then you're in a whole other ballgame. In that case, consider how many more compute nodes would be needed to fit the job in memory (it's a parallel job afterall, hopefully it will scale out this far). RAM is faster than SSDs anyway, so extra money on expensive SSDs would often be better spent on more compute nodes with more RAM.

Jared
  • 519
  • 1
  • 4
  • 11