Dead Lock In a Single Producer Multiple Consumer Case

Question

Could anyone point out why this code can cause dead-lock? It is a single producer, multiple consumer problem. The producer have 8 buffers. Here it has 4 consumers. Each consumer will have two buffers. When a buffer is filled, it flags it to be ready to consume and switch to the second buffer. The consumer then can process this buffer. After it done, it return the buffer to the producer.

Buffer 0-1 for consumer 0 Buffer 2-3 for consumer 1 Buffer 4-5 for consumer 2 Buffer 6-7 for consumer 3

The program once a while reaches to a dead lock state. The understanding is that, since the flag can be only in one state, either 0 or 1, so at least either consumer or producer can proceed. It one proceed, it eventually will unlock the dead lock.

#include <iostream>
#include <thread>
#include <mutex>

using namespace std;

const int BUFFERSIZE = 100;
const int row_size = 10000;
class sharedBuffer
{
public:
    int B[8][BUFFERSIZE];
    volatile int B_STATUS[8];
    volatile int B_SIZE[8];
    sharedBuffer()
    {
        for (int i=0;i<8;i++)
        {
            B_STATUS[i] = 0;
            B_SIZE[i] = 0;
            for (int j=0;j<BUFFERSIZE;j++)
            {
                B[i][j] = 0;
            }
        }
    }

};

class producer
{
public:
    sharedBuffer * buffer;
    int data[row_size];
    producer(sharedBuffer * b)
    {
        this->buffer = b;
        for (int i=0;i<row_size;i++)
        {
            data[i] = i+1;
        }
    }
    void produce()
    {
        int consumer_id;
        for(int i=0;i<row_size;i++)
        {
            consumer_id = data[i] % 4;
            while(true)
            {
                if (buffer->B_STATUS[2*consumer_id] ==1 && buffer->B_STATUS[2*consumer_id + 1] == 1)
                continue;
                if (buffer->B_STATUS[2*consumer_id] ==0 )
                {
                    buffer->B[2*consumer_id][buffer->B_SIZE[2*consumer_id]++] = data[i];
                    if(buffer->B_SIZE[2*consumer_id] == BUFFERSIZE || i==row_size -1)
                    {
                        buffer->B_STATUS[2*consumer_id] =1;
                    }
                    break;  
                }
                else if (buffer->B_STATUS[2*consumer_id+1] ==0 )
                {
                    buffer->B[2*consumer_id+1][buffer->B_SIZE[2*consumer_id+1]++] = data[i];
                                        if(buffer->B_SIZE[2*consumer_id+1] == BUFFERSIZE || i==row_size -1)
                                        {
                                                buffer->B_STATUS[2*consumer_id+1] =1;
                                        }
                                        break;
                } 
            }       
        }
        //some buffer is not full, still need set the flag to 1
        for (int i=0;i<8;i++)
        {
            if (buffer->B_STATUS[i] ==0 && buffer->B_SIZE[i] >0 )
                buffer->B_STATUS[i] = 1;
        }
        cout<<"Done produce, wait the data to be consumed\n";
        while(true)
        {
            if (buffer->B_STATUS[0] == 0 && buffer->B_SIZE[0] == 0 
                && buffer->B_STATUS[1] == 0 && buffer->B_SIZE[1] == 0 
                && buffer->B_STATUS[2] == 0 && buffer->B_SIZE[2] == 0 
                && buffer->B_STATUS[3] == 0 && buffer->B_SIZE[3] == 0
                && buffer->B_STATUS[4] == 0 && buffer->B_SIZE[4] == 0 
                && buffer->B_STATUS[5] == 0 && buffer->B_SIZE[5] == 0 
                && buffer->B_STATUS[6] == 0 && buffer->B_SIZE[6] == 0 
                && buffer->B_STATUS[7] == 0 && buffer->B_SIZE[7] == 0 )             
            {
                for (int i=0;i<8;i++)
                    buffer->B_STATUS[i] = 2;
                break;
            }
        }       
    };

};

class consumer
{
public:
    sharedBuffer * buffer;
    int sum;
    int index;
    consumer(int id, sharedBuffer * buf){this->index = id;this->sum = 0;this->buffer = buf;};
    void consume()
    {
        while(true)
        {
            if (buffer->B_STATUS[2*index] ==0 && buffer->B_STATUS[2*index+1] ==0 )
                continue;
            if (buffer->B_STATUS[2*index] ==2 && buffer->B_STATUS[2*index+1] ==2 )
                                break;
            if (buffer->B_STATUS[2*index] == 1)
            {
                for (int i=0;i<buffer->B_SIZE[2*index];i++)
                {
                    sum+=buffer->B[2*index][i];
                }
                buffer->B_STATUS[2*index]=0;
                buffer->B_SIZE[2*index] =0; 
            }

            if (buffer->B_STATUS[2*index+1] == 1)
                        {
                                for (int i=0;i<buffer->B_SIZE[2*index+1];i++)
                                {
                                        sum+=buffer->B[2*index+1][i];
                                }
                                buffer->B_STATUS[2*index+1]=0;
                                buffer->B_SIZE[2*index+1] =0;
                        }

        }
        printf("Sum of consumer %d = %d \n",index,sum);
    };

};
int main()
{
    sharedBuffer b;
    producer p(&b);
    consumer c1(0,&b),c2(1,&b),c3(2,&b),c4(3,&b);
        thread p_t(&producer::produce,p);
    thread c1_t(&consumer::consume,c1);
    thread c2_t(&consumer::consume,c2);
    thread c3_t(&consumer::consume,c3);
    thread c4_t(&consumer::consume,c4);
    p_t.join();c1_t.join();c2_t.join();c3_t.join();c4_t.join();
}

score 0 · Answer 1 · answered Jul 23 '15 at 23:01

This is flawed in many ways. The compiler can reorder your instructions, and different CPU cores may not see memory operations in the same order.

Basically your producer does this:

it writes data to the buffer
it sets the flag

Your consumer does this:

it reads the flag
if the flag is what it wants it reads data
it resets the flag

This does not work, for several reasons.

The compiler can reorder your instructions (both on the consumer and producer side) to do things in a different order. For example, on the producer side, it could store all your computations in registers, and then write the status flag to memory first, and the data later. The consumer would then get stale data.
Even in absence of that, there is no guarantee that different writes to memory are seen in the same order by different CPU cores (e.g. if they have separate caches, and your flag and data are on different cache lines).

This can cause all sorts of trouble - data corruption, deadlocks, segfaults, depending on what exactly your code does. I haven't analyzed your code sufficiently to tell you exactly why this causes a deadlock, but I'm not surprised at all.

Note that the 'volatile' keyword is completely useless for this type of synchronization. 'volatile' is only useful for signal handling (unix signals), not for multithreaded code.

The correct way to do this is to use proper synchronization (for example mutexes) or atomic operations (e.g. std::atomic). They have various different guarantees that make sure that the issues above don't happen.

Mutexes are generally easier to use if speed is not of the highest importance. Atomic operations can get you a little more control but they are very tricky to use.

I would recommend that you do this with mutexes, then profile the program, and then only go to atomic operations if it's insufficiently fast.

valgrind is a great tool which is useful to debug multithreaded programs (it'll point out unsynchronized memory access and the like).

score 0 · Answer 2 · answered Jul 24 '15 at 05:13

0

thanks for the helpful comments. I thought if make sure all the flags/status value are read from memory, not from registers/cache, the deadlock should not happen no matter how compiler reorganize the instructions. The volatile keyword should enforce this. Looks like my understanding is wrong.

Another baffling thing is that, I thought the value of status variable should only be one of (0,1,2), but once a while, I saw the value like 5384. Somehow the data got corrupted.

answered Jul 24 '15 at 05:13

Jack

1

I read some literature about volatile. It is indeed not suitable in a multiple threads case. After changing the volatile to atomic variable in C++11. The deadlock and data corruption go away. atomic B_STATUS[8]; atomic B_SIZE[8]; – Jack Jul 24 '15 at 18:38

Dead Lock In a Single Producer Multiple Consumer Case

2 Answers2