POSIX pthread_join hangs after thousands of threads

Question

I am new to POSIX and I cant find a solution to this particular problem.

Are there any known issues with creating pthreads inside of loop with big number of iterations(>100000)?

It seems like every time I execute, it hangs on a random pthread_join.

I've tested for memory leaks and thread stack usage with valgrind.
If I interrupt gdb when the program hangs it will trace the problem to pthread_join.

This is example code recreates my problem.

    #include <stdio.h>
    #include <stdlib.h>
    #include <pthread.h>

    void* task(void* args);
    int main(int argc, char **argv)
    {
        int num_threads = 4;
        int m = 100000;
        int i;
        for(i = 0; i < m; i++)
        {
            fprintf(stdout, "i:%d\n",i);
            //parallel region starts
                pthread_t threads[num_threads];
                int t,rc;

                for(t = 0; t < num_threads; t++)
                {
                    rc = pthread_create(&threads[t],NULL,task,NULL);
                    if(rc){
                        fprintf(stderr,"ERROR; return code from pthread_create() is %d\n", rc);
                        exit(-1);
                    }
                }
                for(t = 0; t < num_threads; t++)
                {
                    rc = pthread_join(threads[t],NULL);
                    if(rc){
                    fprintf(stderr,"ERROR; return code from pthread_join() is %d\n", rc);
                        exit(-1);
                    }
                }
            //parallel region ends
        }
        pthread_exit(NULL);
    }

    //thread task
    void* task(void* args)
    {
        pthread_exit(NULL);
    }

May be its not hanged, but gone incredibly slow due to starting too many threads — Pras, Mar 27 '18 at 11:28
Yes possible it is just taking a long time. You are creating and destroying 4,000,000 threads here! What is the output from your `fprintf(stdout, "i:%d,j:%d\n",i,j);` - how far does it get before hanging? — GrahamS, Mar 27 '18 at 11:31
@GrahamS , it varies from i:1 , j:xxx to i: 100,j:xxx. Also it is worth mentioning that the task in task manager, stops using cpu time and memory when it hangs. — Edward Ep, Mar 27 '18 at 11:43
Have you tried it with just a single loop from 0 to 1000000? If you still see the problem then that at least lets you remove "nested loop" from your question. It may get more attention if it is a more general issue. — GrahamS, Mar 27 '18 at 12:14
@GrahamS, tested with single loop, still gets stuck at i: 4000-20000 — Edward Ep, Mar 27 '18 at 12:34
Okay so edit your question and code to remove the "nested loop" part as that is a red herring. Change the title to something a bit more general like _"POSIX pthread_join hangs after thousands of threads"_ — GrahamS, Mar 27 '18 at 12:37
'Are there any known issues with creating pthreads inside of loop with big number of iterations' - yes, it's slow, dangerous and avoidable. — Martin James, Mar 27 '18 at 15:26
@MartinJames care to clarify why? Nothing the OP is doing seems particularly wrong. — GrahamS, Mar 27 '18 at 19:50
@MartinJames ??? OP just created 4 threads and join them, in a loop. What is the problem? — Jean-Baptiste Yunès, Mar 28 '18 at 08:14

score 1 · Accepted Answer · answered Mar 27 '18 at 11:45

1

Verified your code on an Ubuntu machine i7-4600U CPU @ 2.10GHz I don't see any issue with your code, just to verify i compiled and run your code on my local machine it seems to work fine.

i got the output as expected:

i:999,j:999

so I am assuming the issue you you are experiancing has something to do with your environment. what CPU OS and GCC are you using?

answered Mar 27 '18 at 11:45

Omer Dagan

662
5
14

Tested on Ubuntu bash on windows 10 machine Phenom 2 X4 @ 3.4Ghz and Ubuntu machine i5-4210U @ 1.7Ghz. Both run gcc v. 5.4.0 and ubuntu dist. 16.04. – Edward Ep Mar 27 '18 at 12:17
Your assumption was right. It was the ubuntu bash on windows 10 that was creating the problem. I run the code on a VM and it worked. – Edward Ep Mar 28 '18 at 17:28

POSIX pthread_join hangs after thousands of threads

1 Answers1