How to allow certain threads to have priority in locking a mutex use PTHREADS

Question

Assume that the following code is being executed by 10 threads.

pthread_mutex_lock(&lock)
Some trivial code
pthread_mutex_unlock(&lock)

For purpose of explanations lets say the threads are T1, T2, T3.....T10. My requirement is that as long as T1 or T2 or T3( i.e any of T1, T2 or T3) is waiting for acquiring a lock, the other threads i.t T4, T5, T6.....T10 should not be able to acquire the lock i.e T1, T2 and T3 should have precedence in acquiring the lock with respect to other threads.

I guess it could be done by increasing the priority of threads T1, T2 and T3

i.e here is the pseudo code

if this thread is T1 or T2 or T3
increase its priority 
pthread_mutex_lock(&lock)
Some trivial code
pthread_mutex_unlock(&lock)
if this thread is T1 or T2 or T3 decrease it priority to normal

Please note that I want a solution that is works for Linux platform and should use pthreads. I don't really care about any other platform.

Also note that I don't really want to make these 3 threads as realtime, I want them to exhibit their defualt behaviour(scheduling and priority) except that in the above mentioned small piece of code I want them to always have precedence in acquiring lock.

I have read some man pages about scheduling policies and scheduling priorities in Linux but can't really make out :(

Will this work? Can you help me with the exact pthread API required to accomplish the above task?

Regards lali

Answer 1

Here's my implementation. Low priority threads use prio_lock_low() and prio_unlock_low() to lock and unlock, high priority threads use prio_lock_high() and prio_unlock_high().

The design is quite simple. High priority threads are held at the critical section mutex ->cs_mutex, low priority threads are held at the condition variable. The condition variable mutex is only held around updates to the shared variable and signalling of the condition variable.

#include <pthread.h>

typedef struct prio_lock {
    pthread_cond_t cond;
    pthread_mutex_t cv_mutex; /* Condition variable mutex */
    pthread_mutex_t cs_mutex; /* Critical section mutex */
    unsigned long high_waiters;
} prio_lock_t;

#define PRIO_LOCK_INITIALIZER { PTHREAD_COND_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER }

void prio_lock_low(prio_lock_t *prio_lock)
{
    pthread_mutex_lock(&prio_lock->cv_mutex);
    while (prio_lock->high_waiters || pthread_mutex_trylock(&prio_lock->cs_mutex))
    {
        pthread_cond_wait(&prio_lock->cond, &prio_lock->cv_mutex);
    }
    pthread_mutex_unlock(&prio_lock->cv_mutex);
}

void prio_unlock_low(prio_lock_t *prio_lock)
{
    pthread_mutex_unlock(&prio_lock->cs_mutex);

    pthread_mutex_lock(&prio_lock->cv_mutex);
    if (!prio_lock->high_waiters)
        pthread_cond_signal(&prio_lock->cond);
    pthread_mutex_unlock(&prio_lock->cv_mutex);
}

void prio_lock_high(prio_lock_t *prio_lock)
{
    pthread_mutex_lock(&prio_lock->cv_mutex);
    prio_lock->high_waiters++;
    pthread_mutex_unlock(&prio_lock->cv_mutex);

    pthread_mutex_lock(&prio_lock->cs_mutex);
}

void prio_unlock_high(prio_lock_t *prio_lock)
{
    pthread_mutex_unlock(&prio_lock->cs_mutex);

    pthread_mutex_lock(&prio_lock->cv_mutex);
    prio_lock->high_waiters--;
    if (!prio_lock->high_waiters)
        pthread_cond_signal(&prio_lock->cond);
    pthread_mutex_unlock(&prio_lock->cv_mutex);
}

Answer 2

As I understand it, the only way you can truly guarantee this would be to write a lock that works like that yourself. However @xryl669's answer that suggests using thread priority and priority inheritance is certainly worthy of consideration if it works for your use case.

To implement it yourself, you will need condition variables and counts of the number of waiting low / high priority threads.

In terms of the concepts and APIs you'll need, it is relatively similar to implementing a read/write lock (but the semantics you need are completely different, obviously - but if you understood how the r/w lock is working, you'll understand how to implement what you want).

You can see an implementation of a read write lock here:

http://ptgmedia.pearsoncmg.com/images/0201633922/sourcecode/rwlock.c

In the lower priority threads, you'd need to wait for high priority threads to finish, in the same way readers wait for writers to finish.

(The book the above code is taken from it also a great posix threads book btw, http://www.informit.com/store/product.aspx?isbn=0201633922 )

Answer 3

The native way is to enable priority inheritance for your mutex (with pthread_mutex_attr), and use pthread's thread priority to perform what you need. It only requires very few lines of code, and you are not re-inventing the wheel. On the good side, it'll also work with RT or FIFO scheduler while your homebrew version will not.

Then, whenever a thread with a high priority waits on a mutex that's acquired by a thread on lower priority, the kernel "boost" the low priority thread so it can be scheduled in place of the high priority thread, thus giving it a timeslice to release the lock. As soon as the lock is released, the high priority thread is scheduled. That's the lowest delay you could get since it's done in the kernel.

Answer 4

Alternatively you may just introduce another lock for higher priority threads. consider the following pseudo-code (i am not familiar with the pthread semantics, but i believe this is not hard to map the code to the needed calls)

EDIT (thanx JosephH)

introducing the exec semaphore set to 3 (number of high-prio threads) note that pend(exec,3); means that this pend will sleep until all 3 slots are available and will consume them all

//init
exec = semaphore(3,3);

//========================

if this is NOT thread (t1,t2,t3)
    lock(low_prio);
    sem_pend(exec,3);
else
    sem_pend(exec,1);
lock(high_prio);
//...
unlock(high_prio);
if this is NOT thread (t1,t2,t3)
    sem_release(exec,3);
    sleep(0); //yield();  //ensures that sem_pend(exec,1) is executed
    unlock(low_prio);
else
    sem_release(exec,1);

Answer 5

(The first two attempts had bugs, pls jump to EDIT2)

Maybe this would work?

if NOT this thread is T1 or T2 or T3
    pthread_mutex_lock(&lock1) // see note below
    pthread_mutex_lock(&lock2)
    Some trivial code
    pthread_mutex_unlock(&lock2)
    pthread_mutex_unlock(&lock1)
else
    pthread_mutex_lock(&lock2)
    Some trivial code
    pthread_mutex_unlock(&lock2)        
end if

Reasoning: Some threads will compete for two locks and therefore will have lower priority and some threads will compete for only one lock and therefore will have higher priority. Still the difference might be marginal and then the resolution would be to introduce some lag between acquiring first lock and attempting the second lock for the higher priority threads in which time the higher priority threads will be given a chance to get the lock2.
(disclaimer: I am a newbie when it comes to this)

EDIT: Another attempt/approach

if NOT (this thread is T1 or T2 or T3)  
    pthread_mutex_lock(&lock1)
    if pthread_mutex_trylock(&lock2) == 0  // low priority threads will not get queued
        Some trivial code
        pthread_mutex_unlock(&lock2)
    end if
    pthread_mutex_unlock(&lock1)
else 
    if (this thread is T1 or T2 or T3)
        pthread_mutex_lock(&lock2)
        Some trivial code
        pthread_mutex_unlock(&lock2)        
    end if
end if

EDIT2: Another attempt (trying to learn something here)

if NOT (this thread is T1 or T2 or T3)  
    pthread_mutex_lock(&lock1)
    while !(pthread_mutex_trylock(&lock2) == 0)
        pthread_yield()
    Some trivial code
    pthread_mutex_unlock(&lock2)
    pthread_mutex_unlock(&lock1)
else 
    if (this thread is T1 or T2 or T3)
        pthread_mutex_lock(&lock2)
        Some trivial code
        pthread_mutex_unlock(&lock2)        
    end if
end if

Answer 6

To implement that with pthreads you would need N lists, one per thread priority. The lists would contain pointers to the thread's pthread_cond_t variables.

Schematic untested meta-code:

/* the main lock */
pthread_mutex_t TheLock = PTHREAD_MUTEX_INITIALIZER;

/* service structures: prio lists and the lock for them */
pthread_mutex_t prio_list_guard = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t *prio_lists[MY_MAX_PRIO][MY_MAX_THREAD]; /* 0 == highest prio */

/* lock */
void
prio_lock(int myprio)
{
    pthread_cond_t x;

    pthread_mutex_lock( &prio_list_guard );

    if (0 == pthread_mutex_trylock( &TheLock )) {
        pthread_mutex_unlock( &prio_list_guard );
        return 0;
    }

    pthread_cond_init( &x, 0 );
    LIST_ADD( prio_lists[myprio], &x )

    while(1)    /* handle spurious wake-ups */
    {
        pthread_cond_wait( &prio_list_guard, &x );
        if (0 == pthread_mutex_trylock( &TheLock )) 
        {
            LIST_REMOVE( prio_lists[myprio], &x );
            pthread_mutex_unlock( &prio_list_guard );
            return 0;
        }
    }
}

/* unlock */
void
prio_unlock()
{
    int i;
    pthread_cond_t *p;

    pthread_mutex_lock( &prio_list_guard );

    for (i=0; i<MY_MAX_PRIO; i++)
    {
        if ((p = LIST_GETFIRST( prio_lists[i] )))
        {
            pthread_cond_signal( p );
            break;
        }
    }

    pthread_mutex_unlock( &TheLock );

    pthread_mutex_unlock( &prio_list_guard );
}

The code also handles spurious wake-ups from pthread_cond_wait(), but frankly I have never seen that happening.

Edit1. Note that prio_lists above is a primitive form of a priority queue.