I understand what priority inheritance is. I also understand, from the Mars Pathfinder's system reset issue, that most of the time, depending upon the criticality of the operation, it is good to enable/implement priority inheritance.
However, are there any cases where priority inheritance is not desirable and may actually cause problems if enabled/implemented? If so, could you please provide an example while preferably describing a problem?
OP gives an example where priority inheritance is desirable. There we have a short mutex-protected transaction between threads of highest and lowest priorities and a long-running thread with medium priority which could block the transaction if priority inheritance is not used. Also in this example high-priority thread is more important for the application than medium-priority thread.
To get priority inheritance not desirable, we could make an application where all these assumptions are opposite to the above example. Let's make transaction between threads of highest and lowest priorities longer than time available for medium-priority thread to perform its task. And let's assume that the results of this medium-priority thread are more important than the results of high-priority thread.
Imagine an application which should serve 100 interrupts per second with its high-priority thread. And 10 interrupts per second with its medium-priority thread (where each interrupt needs 30 ms to be processed). Low-priority thread could lock some resource (used together with high-priority thread). And sometimes (very rarely) it could lock it for a long time (1 sec). With priority inheritance first access to this resource by high-priority thread boosts priority of background thread for up to 1 sec. So that medium-priority thread would miss 10 interrupts. At the same time high-priority thread could miss 100 interrupts. Without priority inheritance medium-priority thread serves all interrupts but high-priority thread misses up to 130 interrupts. If interrupts served by medium-priority thread are more valuable, we should prefer to disable priority inheritance.
I never seen a real-life application like this. So I invented one to illustrate this case. Let it be a video capture application with some computer vision task in background and (as additional bonus) sound recording. Video frames are captured by medium-priority thread. Sound is captured by high-priority thread (because otherwise video capturing process could block significant portion of sound interrupts). Video is captured to pre-allocated buffer. Audio is captured with silence suppression (no memory is needed to capture silence). So audio thread needs dynamically allocated memory blocks (this is our shared resource). Computer vision task also sometimes allocates memory blocks. When we are out-of-memory, garbage collector thread blocks memory allocation and does some work. Here without priority inheritance video capture works flawlessly. But with priority inheritance audio thread sometimes blocks video capture (which is considered bad for this application).
Also there are applications where priority inheritance does not give any benefits:
If priority inheritance is enabled in such cases, we could only get some performance (or power efficiency) degradation because priority inheritance implementation in kernel most likely needs some additional resources.
