3

I'm using synchronization KPI in my macOS kernel extension to verify that one function is fully performed before the other function starts (both functions are performed in different threads of course).

These are the synchronization methods : 

msleep(void *channel,lck_mtx_t *mtx,int priority,const char *wmesg,  struct  timespec *timeout);
wakeup(void *channel);

So channel is a pointer to boolean value that represent first function was fully performed.

Here's my implementation in the first function:

OSIncrementAtomic(channel);
wakeup(channel);

And on the other function I wait for channel to be set: 

msleep(channel, NULL, 0, "", ts);

However, in case the first function terminated before the second one (which is the common case), I shell wait for the timeout in ts for vein.

My question is whether there's a way to skip the msleep in case the wakeup already occurred ?

thanks,

2 Answers2

2

There are two issues with your use of msleep.

First, channel is an opaque value, not a "pointer to boolean value"; you do not increment it, it will not be modified by the call. Instead, you simply have to ensure that it is unique, i.e. not used by other unrelated calls to msleep. Convention is to use the memory address of a related data structure to achieve uniqueness. If in your case you cannot ensure that msleep is called before wakeup, then you would use a Mach semaphore, not msleep/wakeup.

The other issue is, as @mrdvlpr has already pointed out in a comment, that msleep needs a mutex in order to be able to wake up. If you call it with mtx=NULL, it will sleep indefinitely and calls to wakeup on the same channel will not have any effect. If you want to wake up later with wakeup or wakeup_once, you need to provide a valid and locked mutex instead of NULL.

A minimal but functional call to msleep without timeout in XNU looks like this:

lck_grp_t *lck_grp;
lck_mtx_t *lck_mtx;

lck_grp = lck_grp_alloc_init("com.example.mslpex", LCK_GRP_ATTR_NULL);
if (!lck_grp)
    /* handle failure */
lck_mtx = lck_mtx_alloc_init(lck_grp, LCK_ATTR_NULL);
if (!lck_mtx)
    /* handle failure */

/* ... */

lck_mtx_lock(lck_mtx);
error = msleep(channel, lck_mtx, pri|PDROP, "mslpex", NULL);
if (error)
    /* handle failure */
else
    /* handle success */

/* ... */

lck_mtx_free(lck_mtx, lck_grp);
lck_grp_free(lck_grp);

The corresponding wake-up call is simply:

wakeup(channel);

In the interest of system stability you probably want to use a timeout on the call to msleep in order to recover from situations where wakeup does not get called as expected.

Daniel Roethlisberger
  • 6,958
  • 2
  • 41
  • 59
  • Hi, I'm also using the msleep/wakeup technic but I used timeout instead of mutex, and I've got the following panic `thread_invoke: preemption_level -1, possible cause: unlocking an unlocked mutex or spinlock` .. do you have any insights about how can it be ? (I've tested it on 10.14 beta). thanks ! – Zohar81 Jun 10 '18 at 14:25
  • Hard so tell without seeing your code. In any case, you might be interested in `IOSleep()` if you are after a simple timed sleep without programmatic wakeup capability. – Daniel Roethlisberger Jun 10 '18 at 15:47
  • I actually need the wakeup capability. The project is based on driver-userspace agents whereas the driver get events and send them to user-space straight away. The driver waits for completion using the `msleep` command and when the userspace finishes, it sends back IOkit command to driver with the matching channel to wakeup the waiting thread. However, there can be multiple requests being handled concurrently so I refrained from using mutex in `msleep`.. maybe I need to have mutex array in the size of the concurrency level. I'd be happy to hear you advice here. thanks again ! – Zohar81 Jun 17 '18 at 07:58
  • And one more thing, do you assume that the `msleep` release the mutex once the `wakeup` on the same channel is received or upon timeout ? thanks – Zohar81 Jun 17 '18 at 10:04
  • I think you may find answers in old FreeBSD manual pages: https://www.freebsd.org/cgi/man.cgi?query=msleep&apropos=0&sektion=0&manpath=FreeBSD+5.5-RELEASE+and+Ports&arch=default&format=html - I am not an expert on in-kernel concurrency so I cannot give you reliable advice on what locking strategy is best for your use case. Lock reacquisition behaviour depends on whether you use `PDROP` in priority or not; I would recommend to look into the relevant XNU source code to check what the exact behaviour of `msleep`. – Daniel Roethlisberger Jun 26 '18 at 08:27
1

Use lck_mtx_sleep_deadline() (or the IOLockSleepDeadline() wrapper) in connection with a mutex. On the thread that waits:

  1. Lock mutex
  2. Check wait condition (is first function still running?)
  3. If yes, call lck_mtx_sleep_deadline()
  4. Unlock mutex

You may want to make steps 2+3 a while() loop depending on your situation.

In your "first function":

  1. Lock mutex
  2. Clear wait condition (increment channel)
  3. Send wakeup to event. (e.g. thread_wakeup_prim((event_t) event, oneThread, THREAD_AWAKENED);)
  4. Unlock mutex.

You can probably also do all of this with the lower-level msleep()/wakeup() directly somehow, but I'm not so familiar with that API.

pmdj
  • 22,018
  • 3
  • 52
  • 103
  • 3
    I think msleep/wakeup is actually built on top of lck_mtx_sleep_deadline() and does exactly what it is being described above. He is just missing an actual mutex lock in the msleep call, he's passing NULL. – mrdvlpr Nov 04 '17 at 21:09
  • @mrdvlper, thanks.. now that's also an option I wasn't aware of, I can pass the mutex to msleep, and it's being passed to lck_mtx_sleep_deadline eventually. –  Nov 05 '17 at 08:12