1

I have a large job and I'm experimenting with the topology to see what gives better results (playing with ntasks, ncores-per-cpu, nodes, etc.) I'm using slurm as my work queue manager.

I have two nodes (CentOS 7, managed with Bright 7), each have 2xIntel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz. So, two processors with 12 cores, all hyper threaded, so I appear to have 48 cores per node. (See below for the processor topology lstopo gives)

My question: should I disable hyperthreading in the bios, or try to disable it in SLURM? Or just treat my system like I have double the cores? Specifying --thread-per-core=1 doesn't seem to have an effect.

My job is a big environmental model, lots of I/O, lots of matrix calculations, etc, and takes on the order of days right now to run.

I've been reading the SLURM FAQ, but am still confused how to proceed.

Processor topology showing NUMA spaces

Matt
  • 171
  • 2
  • 9
  • 3
    What on earth are you talking about? AMD processors do not do hyperthreading. – Michael Hampton Mar 07 '16 at 04:55
  • ACK, I'm dumb. I was running `lscpu` on the headnode, and not on the Intel node. I edited the question to reflect this. (shouldn't have wrote this question at 1 am :( ) – Matt Mar 07 '16 at 16:17
  • 1
    Well then. It really depends entirely on your workload. Put together a job that takes ~ 1 hour or so on the current configuration, and then run it without hyperthreading and see how long it takes. The answer should then be obvious. As a general rule, though, hyperthreading wins for _most_ things. – Michael Hampton Mar 07 '16 at 16:28

2 Answers2

1

Basically, if you feel that your operations would benefit from the freer L2/L3 cache that is possible when HT is disabled, go ahead and disable HT from the BIOS.

If however there is a large number of small operations that can be run in parallel, you are definitely benefiting from leaving HT enabled.

VMWare ESXi and Hyperthreading

http://lifehacker.com/how-hyper-threading-really-works-and-when-its-actuall-1394216262

Community
  • 1
Sam Perry
  • 41
  • 5
1

Slurm allocates by cores, SLURM was designed for portability not performance. So the way it allocates is "Note that even on systems with hyperthreading enabled, the resources will generally be allocated to jobs at the level of a core (see NOTE below). Two different jobs will not share a core.". It does know what hyperthreading is and it can be enabled but its specifically looking at cores per socket. It is open source and I'm sure a hyperthread optimization is in order.

Jim B
  • 24,081
  • 4
  • 36
  • 60
  • I read that, but I was still having trouble making it target two nodes if it felt that one node wasn't full. Maybe htop isn't the best tool to measure how it's running. Reading this, and the previous answer, I think that I really just need to keep experimenting with different combinations. I'm getting the impression that hyper-threading is helping me out though, but still am not sure. – Matt Mar 07 '16 at 19:12
  • It might be, from slurms point of view its ignoring it, and if you are using slurm, performance isn't the goal anyway. – Jim B Mar 07 '16 at 19:48
  • What do you mean by that? I was under the impression that slurm shouldn't impact performance if set up properly. Right now my stuff is taking way too long so performance is key. I'm trying to run a model (government written Fortran model) whose setup scripts require a workqueue manager, and slurm is the one our cluster came with.... Would a different queue make a big difference? – Matt Mar 08 '16 at 02:20
  • 1
    it could, Univac, is my recommendation when performance is required, it has phi support and gpu support, and CPU scavenging. requests for product recommendations are off topic however. – Jim B Mar 08 '16 at 08:50