0

I faced a concurrency problem when writing to the same named pipe created with mkfifo by multiple processes at the same time, where some writes got lost. Since the number of writing processes are limited I want to switch from "writing to 1 pipe from n processes and reading from 1 separate" to "writing to n pipes by n processes and reading from 1 separate process".

Currently I'm reading via read line <"$pipe" in a loop until a condition is met. read blocks here until a line was read.

How can I read from multiples pipes ($pipe1, $pipe2 … $pipeN) via one loop until a condition is met, while honouring newly written lines on all pipes the same?

Sebastian Barth
  • 4,079
  • 7
  • 40
  • 59
  • 2
    The notation you're using `read line < "$pipe"` opens and closes the pipe on each iteration. When the (only) reader closes the pipe, all contents are lost — that's why you're missing messages. You need something more like `while read line; do … done < "$pipe"` which keeps the pipe open until the last writer closes it. Using named pipes is tricky. Independently of whether you're using named pipes or not, I/O redirection on the `read` command is usually wrong — unless you're really sure you only need one input. – Jonathan Leffler Nov 08 '20 at 17:12
  • 2
    I think this is probably a duplicate of [How to avoid `echo` closing FIFO named pipes — Funny behaviour of Unix FIFOs?](https://stackoverflow.com/questions/8410439/how-to-avoid-echo-closing-fifo-named-pipes-funny-behavior-of-unix-fifos/8410538#8410538) I think the multiple pipes part of the question is more of an [XY Problem](http://mywiki.wooledge.org/XyProblem) than what you need. You'll need a C program if you do need to read from more than 1 pipe — using `select()` or `poll()` or something similar to find which pipes actually have data to read. – Jonathan Leffler Nov 08 '20 at 17:14
  • Thanks for the explanation of the initial problem. Indeed it is an XY problem. Sadly your hint did not really work out. I posted a [question](https://stackoverflow.com/questions/64743111/writing-from-multiple-processes-launched-via-xargs-to-the-same-fifo-pipe-causes) for that initial problem and included the results of switching to your suggestion. Anyway – even if it won't solve my initial problem – I would really like to know if there is a solution to the problem described in **this** post? – Sebastian Barth Nov 08 '20 at 21:20
  • Please provide a [mre]; it's not really clear what you are asking. – tripleee Nov 09 '20 at 05:43

1 Answers1

0

One way to deal with the initially described problem of multiple children writing to a single FIFO is to have a process open the FIFO for reading but never actually read it. This will allow writers to write unless the FIFO is full. I don't think there's a standard program that simply goes to sleep forever until signalled. I use a home-brew program pause, which is a pretty minimal C program:

#include <unistd.h>

int main(void)
{
    pause();
}

It never exits until signalled. In your shell script, first launch the children, telling them to write to $FIFO, then run:

pause <$FIFO &
pid=$!

Note that pause-like command will not be launched into the background until the redirection completes, and the open of the FIFO won't complete until there is a process to write to the FIFO — so at least one child needs to be launched in background before the pause-like process is executed. Or write a variant of pause (I call mine sleepon) which opens the files named in its argument list — then the command line is similar to sleepon $FIFO & and the backgrounding operation completes and the pause-like program blocks until it is able to open the FIFO (which will be when one of the children opens the FIFO for writing), and then goes to sleep indefinitely. But the code for sleepon is a lot more complex than the code for pause.

Once the children and the pause-like process are launched, the parent can continue with the main processing loop.

while read line
do
    …
done < $FIFO

The main thing to be aware of is that the parent loop will exit whenever the FIFO is emptied. You need to know when it should terminate, if ever. At the point where it does terminate, it should kill the pause process: kill $pid. You may need to wrap a while true; do … done loop around the line-reading loop — but you may need something cleverer than that. It depends, in part, on what your "until a condition is met" requirement is.

Your requirement to 'read from multiple FIFOs, all of which may intermittently have data on them' is not easy to do. It's not particularly trivial in C; I don't think there's a standard (POSIX) shell command to assist with that. In C, you'd end up using POSIX select() or poll() or one of their many variants — some of which are platform-specific. There might be a platform-specific command that will help; I have my doubts, though.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278