1

I wish to add support for CPU hot plugging in my application that has strict affinity rules. Each physical core has exactly one thread pinned to it, but the logic I use for pinning a thread to a CPU is fairly naive and fails if a CPU between 0..N is offline.

I have chosen an approach where I have an array of size MAX_CPU, where each CPU on the system maps to a slot by its identifier. For example, CPU0 -> threads[0], and CPU1 -> threads[1], and so on. The idea is to mirror the system's setup.

for (i = 0; i < N; ++i)
    set_affinity(threads[i], i);

However, if an offline CPU is encountered anywhere but the end, it will fail.

To make things worse, when a CPU is taken offline during runtime, the pinned thread's affinity mask is reset without notice.

Ultimately, I hope to support complex setups like:

CPU0     CPU1      CPU2     CPU3
ONLINE   OFFLINE   ONLINE   OFFLINE

How can I incorporate awareness of online and offline CPU:s into my application?

I'm avoiding /proc and /sys as I'm interested in porting to other platforms, specifically various BSD:s. I use x86_64 for now, so the cpuid instruction may be useful.

haste
  • 1,441
  • 1
  • 10
  • 21
  • "*I'm avoiding /proc and /sys as I'm interested in porting to other platforms*" - How are you currently accessing CPU online/offline status? – ArjunShankar Aug 22 '12 at 18:12
  • @ArjunShankar I can't access per-CPU status, but on Linux I can get the count of online and configured CPU:s using `sysconf(2)`. For now, I just optimistically pin N threads throughout CPU:s 0..N and let it fail for offline CPU:s. What I'm looking for is a reliable way to enforce each online CPU to have a thread pinned to it. – haste Aug 22 '12 at 22:02
  • I don't see proper support for CPU hotplugging in FreeBSD, for example. I could be wrong because the only relevant result I found was [from 2009, and it was in the negative](http://lists.freebsd.org/pipermail/freebsd-current/2009-April/005706.html). – ArjunShankar Aug 22 '12 at 22:15
  • On the other hand, there seems to be pretty good documentation about CPU hotplugging in Linux. [Here is a 2005 announcement on ML/LWN](http://lwn.net/Articles/159561/) about the 'new' documentation. And [here is the document itself](http://www.kernel.org/doc/Documentation/cpu-hotplug.txt) – ArjunShankar Aug 22 '12 at 22:24
  • My point with the above being, I *think* you don't really have a way out but to write this stuff non-portably. I've never tried it, so I could be wrong. Everything I said above is a result of searches (which I did because I found your question interesting). – ArjunShankar Aug 22 '12 at 22:26
  • @ArjunShankar Thank you for your input. I feared it would be very platform specific and hacky. It looks like Linux is the only platform to take seriously on CPU hot plugging for now. This won't be pretty... – haste Aug 22 '12 at 23:43

1 Answers1

0

Turns out sched_getaffinity(2) is awesome and better suited for this case. The cpu_set_t it fills in is not a generalized mask like 0xffffffff... that would imply schedule anywhere, but actually a detailed and up-to-date mask of each online and permitted CPU. Using the CPU_* macros, it's possible to extract how many CPU:s are online and which.

cpu_set_t set;
int i, n;

if (sched_getaffinity(0, sizeof(cpu_set_t), &set))
    return;

n = CPU_COUNT(&set);

/* pin a thread to each online and permitted cpu */
for (i = 0; n; ++i)
    if (CPU_ISSET(i, &set)) {
        /* spawn a thread and pin it to cpu identified by 'i' ... */
        --n;
    }

When notified of a hot plug event, a call to sched_getaffinity(2) from a non-pinned thread will give us an updated mask.

An added benefit is awareness of utilities like taskset(1).

FreeBSD has a cpuset_getaffinity(2) system call that probably operates in a similar fashion. I don't have access to try it out now.

haste
  • 1,441
  • 1
  • 10
  • 21