6

I'm building a topological tree of sockets, NUMA nodes, caches, cores, and threads for any Intel or AMD system in C.

Building this hierarchy, I want to ensure hardware threads are grouped together appropriately so it's clear who precisely shares what. I've found that I can set a thread's affinity and then use the cpuid instruction to get a lot of the info I want, but not all.

If a package/socket has multiple NUMA nodes, how do I get an index of the NUMA node for the current hardware thread? If the NUMA node has multiple L3 caches, how do I get the index?

AMD has something for NUMA node ID in Fn8000_001E_ECX, but I can't find anything comparable for Intel. And nothing re: L3 index for either.

Andrey Semashev
  • 10,046
  • 1
  • 17
  • 27
Will Leiserson
  • 128
  • 5

1 Answers1

3

If a package/socket has multiple NUMA nodes, how do I get an index of the NUMA node for the current hardware thread?

You get this information from ACPI.

Specifically, there's a "System Resource Affinity Table" (SRAT) that contains a list of structures describing which NUMA domain different things (CPUs, memory areas, ...) are in at boot time. For 80x86; you'd parse this list looking for both "Processor Local APIC/SAPIC Affinity Structures" and "Processor Local x2APIC Affinity Structures".

For hot-plug CPUs the table isn't enough (the SRAT won't change when a CPU is inserted or removed after boot), so you might also need to use an ACPI machine language interpreter to execute _PXM objects to obtain current NUMA information. Computers that support hot-plug CPUs is very rare though.

Note that in ACPI "NUMA domain numbers" are excessively large (32 bits) and not guaranteed to be contiguous (e.g. in theory you could have 2 NUMA nodes with the NUMA domain numbers 0x12345678 and 0x9ABCDEF0); which means that you can't use them for array indexes (e.g. if you want to do something like "NUMA_stats[domain].CPU_count++;" it won't be fun). There is also no standard value reserved for "unknown NUMA domain", which is inconvenient for code that determines topology (e.g. you'd need a separate "did/didn't find a valid NUMA domain" flag to keep track).

Brendan
  • 35,656
  • 2
  • 39
  • 66
  • 1
    So for user-space, you'll have to ask the OS somehow, at least for access to that ACPI data. – Peter Cordes Sep 02 '21 at 04:42
  • 1
    @PeterCordes: Yes. Ideally the OS would obtain all of the topology info (and sanity check it, and correct any errata), and provide a nice portable API with complete information (e.g. like `SYSTEM_LOGICAL_PROCESSOR_INFORMATION` in Windows but with more info), and user-space wouldn't ever want to use `CPUID` or ACPI (unnecessary, non-portable, unchecked, uncorrected). I'd even suggest that, in a perfect world, user-space would be banned from using `CPUID` (too easy for malware to detect security vulnerabilities in CPU). Someone needs to write the OS though.. ;) – Brendan Sep 02 '21 at 05:10
  • Neat idea, but there are legit performance reasons for wanting to know the CPU model number, e.g. to choose not to use `pshufb`-based functions on CPUs that have it but where it's slow (K8, Core 2 Conroe, what x264 calls "SlowShuffle" as a CPU feature). Or more generally for `gcc -march=native` setting tuning options. – Peter Cordes Sep 02 '21 at 05:19
  • If you make this hypothetical OS use HW virt to trap every CPUID execution, you can sanitize it some, but I think you'd be hard pressed to give user-space as much info as it needs for every existing program to do all the legit things some user processes currently do, without making it pretty easy to fingerprint a CPU model even based on available features and maybe a microbenchmark or two. Hiding microcode revision could be a thing, though, for processes without some system-info capability / privilege. – Peter Cordes Sep 02 '21 at 05:21
  • @PeterCordes: There's legitimate reasons for an OS to provide (e.g.) a "pshufb is fast/slow" flag or a "micro-arch number" (and still not allow access to the exact stepping/sub-stepping information or micro-code update version number or "software visible work-arounds" flags). There's also plenty of cases where CPUID provides wrong information, "technically correct but bad" info (e.g. that a known buggy feature is supported), doesn't provide information at all, or where the same information is provided in one of many different ways. – Brendan Sep 02 '21 at 11:49
  • @PeterCordes: Note that when I said "Someone needs to write the OS" I meant it more like "the original asker might be writing an OS (and might not be able to rely on any kind of topology API provided by an existing OS)". – Brendan Sep 02 '21 at 11:55
  • Yeah, I'm stuck firmly in user space. It seems like this kind of topological information would be of interest to a lot of performance-sensitive concurrent applications, but they may make assumptions about the homogeneity of the architecture that tend to hold true in most commercial hardware. – Will Leiserson Sep 02 '21 at 13:57