-1

I have access to a group of servers connected with a 1Gb LAN, and each of them has 40+ cores and Ubuntu OS. They all have a common NAS. I installed SLURM on a few of them and configured it so that each server is both a control and a compute node, and the servers are not connected. The required analyses are bioinformatic and are CPU bound but with files in GB. My questions are as following:

  1. This is not a compute cluster, correct? What would be needed to link these servers into a single cluster?
  2. Is it a valid practice to use HPC managers like Slurm on this configuration? How would the data be shared? They do have a common NAS, but running any computations on the NAS directly is very slow compared to the local files.

My ideal solution would pull the files to a local machine (ideally regardless of their location, but NAS could be the common hub) perform the computation and possibly return the output files. Is this an unreasonable request or a solved problem?

Thanks in advance!

Cindy Almighty
  • 123
  • 4

1 Answers1

2

Yes, that is a compute cluster. I have a broad definition, as multiple compute nodes tasked with the same workload. Large build farm with multiple hosts running compute workers qualifies. So do the enormous systems on the TOP500 list.

More importantly, make use of slurm's capability to manage multiple nodes. Otherwise, one controller only managing itself is a fair bit of complexity, for not a lot of features compared to simpler systems. Add all of the nodes, and divide them up into partitions.

Jobs requesting more than one node require the programs to be multiple node aware, as in MPI. For regular unaware programs, just keep the requests to one node.

Storage, you will need to design something to get files to the nodes. Typically the options include:

  • Copy files to each node's local storage. (slurm sbcast)
  • Stripe a distributed storage across many node's disks. (Lustre, Ceph)
  • High performance file share mounted on each node. (All flash array serving NFS)
John Mahowald
  • 32,050
  • 2
  • 19
  • 34
  • First of all, thanks for a great answer. Before I process this part by part, I have two immediate questions: 1) Can sbcast be truly used if the output files need to be returned, since I don't see an "opposite" command? 2) All-flash array is a hardware solution, correct? Is it a commonly used solution or an exotic, lazy and expensive one? (considering it is a simple solution from my naive perspective) – Cindy Almighty Aug 22 '20 at 23:43
  • I only gave storage examples to distinguish the categories, you still have to design how it would work. Presumably, output would go to some file share, or be uploaded somewhere by some script. All flash array just means your fastest SSDs in a single array. On the high end, it can't compete with giant scale-out distributed storage, but you can put an enormous amount of IOPS and bandwidth into one rack. Cost or build versus buy can't be answered, at least not without a lot more context about your budget and how comfortable you are managing commodity storage. – John Mahowald Aug 23 '20 at 15:29