0

Just booted Arch Linux on a Ryzen Threadripper 1950X server that I built and use in my company. Please don't close this question. It is relevant for anyone using Linux on Threadripper and running NUMA-aware software. In fact I've found the answer and would like to post it.

numactl --hardware reports:

available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 0 size: 128876 MB
node 0 free: 2114 MB
node distances:
node   0 
  0:  10

Threadripper is a NUMA platform and the memory latency increases from ~90ns to ~140ns when crossing dies (source: https://www.servethehome.com/amd-epyc-infinity-fabric-latency-ddr4-2400-v-2666-a-snapshot/). I would like NUMA-aware software to apply its optimizations.

Is this a known problem? How can I fix it?

My hardware:

  • Ryzen Threadripper 1950X
  • ASRock Fatal1ty X399 Professional Gaming
  • Samsung DDR4-2400 ECC UDIMM (8 DIMMs)

My software:

  • Linux 4.12.13 (Arch Linux) (various CONFIG_NUMA kernel options enabled)
Aleksandr Dubinsky
  • 147
  • 2
  • 10
  • Check. Your. Bios. Basically: it works as set up. – TomTom Oct 07 '17 at 13:21
  • Dudes, this is a business environment. This is the most cost efficient workstation environment, even though the boards are named gaming. I am buying some of those for Quantitative Analysis work - 16 sores, 64gb ECC memory, for a price intel dreams of. Gamers (who comes up with those names, damn) actually will use Ryzen - easier fast memory and a lot cheaper. ThreadRipper is really workstation level stuff. – TomTom Oct 07 '17 at 13:23
  • Fucking idiots. Really. – Aleksandr Dubinsky Oct 13 '17 at 10:38
  • 1
    The correct answer is to set "Memory Interleave" to "Channel" in the BIOS. Linux then sees multiple NUMA nodes. See https://www.reddit.com/r/Amd/comments/6stemx/this_is_how_ryzen_master_looks_for_threadripper/dlg9i0i/ – Aleksandr Dubinsky Oct 21 '17 at 09:38

1 Answers1

3

Check your Bios.

Threadripper has two work modes. Your bios sets it in Unified Memory Access Mode.

Not using that particualar board - so no ideea how they name the setting.

Note that the platform is quite new - there are a lot of issues with it through the bios. Can be this gets reset or does not work properly on that parciular bios.

TomTom
  • 51,649
  • 7
  • 54
  • 136
  • Thank you for the tip. It looks like the ASRock does not have this option (yet). It's a shame, because I found this to be the best board given its 10GBase-T Ethernet and U.2 port. – Aleksandr Dubinsky Oct 08 '17 at 09:32
  • I wnent with the MSI X399 for our business. I like 2x M.2 and the 10G I can always add - Intel has great cards. – TomTom Oct 08 '17 at 12:57
  • What is the exact name of the setting in MSI's BIOS? Which board do you have? I'm trying to talk to ASRock support, and point out that a competitor has the feature. – Aleksandr Dubinsky Oct 12 '17 at 16:21
  • No idea. I uhave one MSI X399 Carbon board - prototype for the workstations we buy here now. Never even looked for this setting - the bios i still VERY flaky. Does not recognize my 2666 ECC Ram, does not properly wake up from hibernation. Waiting another month to see how it goes. – TomTom Oct 12 '17 at 16:42
  • Oh. Was there another board where you've seen this setting? Another platform? Only references I've found for configuring NUMA on Threadripper is Ryzen Master. https://www.pcper.com/reviews/Processors/AMD-Ryzen-Threadripper-1950X-and-1920X-Review/NUMA-and-UMA-memory-locality-concer – Aleksandr Dubinsky Oct 12 '17 at 18:31
  • Are you using Unbuffered ECC RAM? Normal Registered DIMMs aren't supported. – Aleksandr Dubinsky Oct 12 '17 at 18:32
  • I found this comment by AMD rep: "changing to "Local Mode" memory enables NUMA by setting Memory Interleaving to Channel Interleaving in the BIOS. Going to "Distributed Mode" enables UMA by setting Memory Interleaving to AUTO/Die Interleaving in the BIOS." I have this setting. https://www.reddit.com/r/Amd/comments/6stemx/this_is_how_ryzen_master_looks_for_threadripper/dlg9i0i/ – Aleksandr Dubinsky Oct 12 '17 at 18:45
  • @AleksandrDubinsky Corsair has ECC - 32gb, 4 pices, 2666 speed. Use 2 of those kits for a 64gb RAM setup - CT4K8G4WFD8266.is the part number. You pretty much can not get them through retail or distributors - they have a webshop where you can order them. – TomTom Oct 13 '17 at 10:41