CPU isolation from Linux scheduler (isolcpus) doesn't work with pthreads

Question

I have Linux with 6 CPU cores, one of which (0st) is isolated from Linux scheduler through the grub config file(/etc/default/grub) with GRUB_CMDLINE_LINUX_DEFAULT="quiet isolcpus=0 nohz_full=1".
I checked isolation with cat /sys/devices/system/cpu/isolated and it's ok.

Here are the full code snippet (compile with gcc -pthread ...):

#define _GNU_SOURCE
#include <stdio.h>
#include <pthread.h>
#include <sched.h>
#include <unistd.h>

void *thread(void *ptr)
{
    for(;;) {
        show_curr_cpu("pthread");
        sleep(2);
    }
}

void show_curr_cpu(char *str)
{
    unsigned int cpu, node;    
    if (getcpu(&cpu, &node) != 0) {
        perror("getcpu");
        return;
    }
    printf("[%s]Current CPU: %u\n", str, cpu);
}

int main(void)
{
    show_curr_cpu("main-bfr");

    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(0, &cpuset); /* 0st CPU core */
    pthread_t current_thread = pthread_self();
    pthread_setaffinity_np(current_thread, sizeof(cpu_set_t), &cpuset);

    show_curr_cpu("main-aft");

    pthread_t thread1;
    int thr = 1;
    if (pthread_create(&thread1, NULL, *thread, (void *) thr)) {
        perror("pthread_create");
        return -1;
    }

    for(;;) {
        show_curr_cpu("main");
        sleep(2);
    }
}

The result of isolation supposes that it

Remove the specified CPUs, as defined by the cpu_number values, from the general kernel SMP balancing and scheduler algroithms

Also:

The isolcpus kernel parameter designates a set of CPUs where the process scheduler will ignore those CPUs. That is to say, the process scheduler will not include those CPUs as targets to put processes on, migrate processes onto or off of, or take a process off a CPU.

But after running my code I see the following strangeness:

[main-bfr]Current CPU: 5
[main-aft]Current CPU: 0
[main]Current CPU: 0
[pthread]Current CPU: 0
[main]Current CPU: 0
[pthread]Current CPU: 0
^C

I.e. pthread runs on the same CPU core. But how does it possible to have two different processes (pthread is a regular process from the Linux kernel point of view) on the same CPU core working well (core sharing) while Linux scheduler does not participate in this?

In the Linux implementation of `pthread_create()`, the new thread inherits a copy of the calling thread's CPU affinity mask (ref: [man page](https://man7.org/linux/man-pages/man3/pthread_create.3.html)). — Ian Abbott, Jun 28 '23 at 15:01
@IanAbbott Thanks, this is quite obvious. But the question was about how two processes can coexist (be scheduled) on one CPU core without *"general kernel SMP balancing and scheduler algroithms"* — red0ct, Jun 29 '23 at 10:40
@redOct *"The only way to move a process onto or off an "isolated" CPU is via the CPU affinity syscalls."* does not imply that there can be only one thread scheduled to run on the isolated CPU. — Ian Abbott, Jun 29 '23 at 12:04
@IanAbbott If I understand correctly what follows from your comment, the phrase ***"Remove the specified CPUs, as defined by the cpu_number values, from the general kernel SMP balancing and scheduler algroithms"*** is misleading and basically incorrect. I.e. scheduler algroithms can be applied on this isolated CPU core to schedule several tasks? — red0ct, Jun 29 '23 at 12:30
@IanAbbott A bit updated question with RedHat's explanation and link. Seems that phrase *"the process scheduler will not include those CPUs as targets to put processes on, migrate processes onto or off of, or take a process off a CPU"* contradicts your comment. — red0ct, Jun 29 '23 at 13:03
The RedHat explanation is a bit better, but you need to read all of it. — Ian Abbott, Jun 29 '23 at 13:08
@IanAbbott You mean *"A process can also put itself onto a CPU listed in isolcpus with scheduling-specific functions such as sched_setaffinity()"*. It's absolutely unclear and doesn't explain how these (e.g. 2) processes co-exist without scheduler algorithms. Can you explain? — red0ct, Jun 29 '23 at 13:17
I admit the text in the "Nutshell" book is absolutely unclear, but nothing in the RedHat clarification *"That is to say, the process scheduler will not include those CPUs as targets to put processes on, migrate processes onto or off of, or take a process off a CPU."* precludes the scheduler scheduling time slices on isolated CPUs for any processes/threads (actually "tasks" in kernel-speak) that have been affined to the isolated CPUs by other means. — Ian Abbott, Jun 29 '23 at 13:51
It may be helpful to interpret that in terms of [the kernel docs' general description of that parameter](https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html?highlight=isolcpu): "Isolate a given set of CPUs from disturbance". The scheduling implications of that are that the kernel will not schedule time for user processes on the specified CPUs except as those processes have affinity for those CPUs. That's it. Processes that *are* scheduled on those CPUs will not be disturbed on those CPUs by processes that do not have affinity for them. — John Bollinger, Jun 29 '23 at 14:22

score 2 · Answer 1 · answered Jun 29 '23 at 02:47

2

But how does it possible to have two different processes (pthread is a regular process from the Linux kernel point of view) on the same CPU core working well (core sharing) while Linux scheduler does not participate in this?

You seem to be reading too much into what it means to isolate CPUs from the kernel's process scheduler.

No thread of any process runs on any core without the kernel scheduling it there. A core being isolated from the process scheduler really means little more than that userspace processes / threads have to have affinity for that core to be scheduled there. It does not mean that the kernel cannot or does not manage the scheduling of all the threads with affinity for that core. Linux does time-sharing just fine.

answered Jun 29 '23 at 02:47

John Bollinger

160,171
8
81
157

*"Linux does time-sharing just fine"* - time-sharing is a part of scheduler algroithms. So seems that phrase *"Remove the specified CPUs, as defined by the cpu_number values, from the general kernel SMP balancing and scheduler algroithms"* is incorrect – red0ct Jun 29 '23 at 12:33
Updated the question with RedHat's explanation and link. Seems that your answer also (as Ian Abbotts's comment) contradicts the phrase *"the process scheduler will not include those CPUs as targets to put processes on, migrate processes onto or off of, or take a process off a CPU"* – red0ct Jun 29 '23 at 13:05
@red0ct, again, no process runs anywhere on the system without the kernel putting it there. And having put one or more processes on a given CPU, yes, it manages their scheduling. The explanation you quote must therefore be understood either as an oversimplification, or as pertaining to the "process scheduler" as just one component of a broader scheduling system. – John Bollinger Jun 29 '23 at 13:24
Sorry and with respect to your skills and experience, but it turns out some kind of esoteric around scheduler. No one here can say exactly how it works with isolated CPU cores, but stated that works. Given that your answer objectively goes against the various documentation that can be found in web (am I wrong?). If you can give accessible explanation what's wrong with the docs and how can it work, I would be very grateful! So far I consider this is a bug in the Linux kernel, because the behavior does not match the documentation (including kernel.org). – red0ct Jun 29 '23 at 13:55
1

@red0ct, I just gave you *two* accessible explanations of what may be wrong with the docs you're referencing: (1) they are an oversimplification, or (2) what they mean by "process scheduler" is something other than the whole scheduling subsystem (so they are misleading). Only the author of those docs or whatever source they drew on themselves can distinguish between these alternatives, but they can be generalized as "those docs lend themselves to an interpretation that is contrary to fact and logic". – John Bollinger Jun 29 '23 at 14:05
1

@red0ct, you might want to read [the actual kernel documentation](https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html?highlight=isolcpu), instead of trying to rely on summaries. That may still not be entirely clear, but where they say "Isolate from the general SMP balancing and scheduling algorithms," you need to understand that that's not the same thing as "disable all scheduling". – John Bollinger Jun 29 '23 at 14:14

CPU isolation from Linux scheduler (isolcpus) doesn't work with pthreads

1 Answers1