Why does pthread_join crash (seg fault) at 306 Joins?

Question

I'm playing around with creating a BUNCH of threads, just see learn a bit more about pthread. I've copied a program I found, and instead of making 1 thread, and joining it, it will create X threads, and join them.

When I tried 100 threads (MAX_THREAD 100), it worked fine. So I tried 1000 threads, and it crashed. Then 500 threads, and it crashed. Loading it into GDB showed it crashed at 306 threads. However, it's inconsistent. I can set it to 304 on the system I'm using, and sometimes it'll crash, sometimes it wont.

I'm keeping all the pthread_t's from the pthread_create in an array, so I shouldn't be trying to join the same thread twice.

The segfault happens here: Program received signal SIGSEGV, Segmentation fault. 0x004e420e in pthread_join () from /lib/libpthread.so.0

Here's the code I'm using.

Code:

#define MAX_THREAD 306

#include <pthread.h>
#include <stdio.h>

int something_worked(void) {
    /* thread operation might fail, so here's a silly example */
    void *p = malloc(10);
    free(p);
    return p ? 1 : 0;
}

void *myThread(void *result)
{
   if (something_worked()) {
       *((int*)result) = 42;
       pthread_exit(result);
   } else {
       pthread_exit(0);
   }
}

int main()
{
   pthread_t tid[MAX_THREAD];
   void *status[MAX_THREAD] ;
   int result[MAX_THREAD];
   int i = 0;
   
    for(i = 0; i < MAX_THREAD; i++)
    {
        pthread_create(&tid[i], NULL, myThread, &result[i]);
    }
    for(i = 0; i < MAX_THREAD; i++)
    {
        pthread_join(tid[i], &status[i]);
    }
    
    for(i = 0; i < MAX_THREAD; i++)
    {
        if (status[i] != 0) {
            printf("Thread:[%d] TID[%02x] result %d\n",i, (unsigned)(tid[i]), result[i]);
        } else {
            printf("thread failed\n");
        }
    }

   return 0;
}

I shouldn't be running out of threads:

cat /proc/sys/kernel/threads-max
7470

ulimit's seem to be "good":

[machine~]$ ulimit -s
10240
[machine~]$ ulimit -v
unlimited

Any idea why my pthread_join crashes?

score 1 · Accepted Answer · answered Jan 23 '22 at 19:58

I shouldn't be running out of threads:

Note that this is a system-wide limit, so you could be running out of threads IF you have some other processes with many threads running.

On my x86_64 system your program runs fine with with 1000 threads.

It does fail when built in 32-bit mode. That is because the default ulimit -s is 8MiB, and each thread inherits that setting. 8MiB * 307 == 2456MiB, which is above 2GiB.

If you are in fact on a 32-bit system, 2GiB might be the memory limit for user-space processes (this depends on how your kernel was configured).

Note that the failure is signaled by error return from pthread_create, and your program does not handle such error return correctly, which is why it crashes.

Specifically:

pthread_t tid[MAX_THREAD];  // uninitialized array of tid's
   
for(i = 0; i < MAX_THREAD; i++)
{
    // If this call fails ...
    pthread_create(&tid[i], NULL, myThread, &result[i]);
    // ... then tid[i] is left as "random garbage" ...
}
for(i = 0; i < MAX_THREAD; i++)
{
    // ... which is used here, resulting in the crash.
    pthread_join(tid[i], &status[i]);
}

ParallelNewB · Answer 2 · 2022-01-24T20:34:50.770

Touche'.

getconf LONG_BIT
32

So you've pointed out 1 error in the code, and 1 error in my assumptions, both of which are correct!
A: Code error is not checking the return value of pthread_create().
B: I assumed I was on a 64-bit machine with plenty of memory.

So I updated the program to check the return value of pthread_create(), and limit the create/join loops to the last "good" create.

#define MAX_THREAD 1000
#include <pthread.h>
#include <stdio.h>

int something_worked(void) {
    /* thread operation might fail, so here's a silly example */
    void *p = malloc(10);
    free(p);
    return p ? 1 : 0;
}

void *myThread(void *result)
{
   if (something_worked()) {
       *((int*)result) = 42;
       pthread_exit(result);
   } else {
       pthread_exit(0);
   }
}

int main()
{
   pthread_t tid[MAX_THREAD] = { };
   void *status[MAX_THREAD] = { };
   int result[MAX_THREAD] = { };
   int i = 0;
   int ptc_rtn = 0;
   int max_t = MAX_THREAD;
   
    for(i = 0; i < MAX_THREAD; i++)
    {
        ptc_rtn = pthread_create(&tid[i], NULL, myThread, &result[i]);
        if( ptc_rtn != 0)
        {
            max_t = i;
            break;
        }
    }
    for(i = 0; i < max_t; i++)
    {
        pthread_join(tid[i], &status[i]);
    }
    
    for(i = 0; i < max_t; i++)
    {
        if (status[i] != 0) {
            printf("Thread:[%d] TID[%02x] result %d\n",i, (unsigned)(tid[i]), result[i]);
        } else {
            printf("thread failed\n");
        }
    }

   return 0;
}

The program now does not segfault, and consistently returns between 302 and 304 pthread_creates, depending on what memory is currently available.

Thanks for finding my error, and the explanation of why it happens where it does!

If the answer resolved your problem, please accept it. It is not necessary to type the corrected program as a separate answer. Also, you could create a lot more threads even on a 32-bit machine -- you just need to explicitly ask for smaller stacks. In your case, 16K stack should let you create at least `300*8*1024/16 -- 153600` threads (though you are likely to run into some other limit first). — Employed Russian, Jan 24 '22 at 21:36
Sorry. I've been a long time watcher, but new to asking. Cheers! — ParallelNewB, Jan 25 '22 at 23:19

Why does pthread_join crash (seg fault) at 306 Joins?

2 Answers2