Poor performance on large writes on O_NONBLOCK fifo in mac os

Question

I have a reader and writer on a FIFO, where the reader must not block indefinitely. To do this, I open the read end with O_NONBLOCK.

The write end can block, so I open it as a regular file. Large writes perform unacceptably awfully - reading/writing a 4MB block takes minutes instead of the expected fraction of a second (expected, because in linux the same code takes a fraction of a second).

Example code in Python replicating the issue. First, create a fifo using mkfifo, e.g. mkfifo some_fifo, then run the reading end, then the writing end.

Reading End:

import os, time
# mkfifo some_fifo before starting python
fd = os.open('some_fifo',os.O_RDONLY | os.O_NONBLOCK)
while True:
    try:
        read = len(os.read(fd, 8192)) # read up to 8kb (FIFO buffer size in mac os)
        print(read)
        should_block = read < 8192 # linux
    except BlockingIOError:
        should_block = True # mac os
    if should_block:
        print('blocking')
        time.sleep(0.5)

Write End:

import os
fd = os.open('some_fifo',os.O_WRONLY)
os.write(fd, b'aaaa'*1024*1024) # 4MB write

Note: The original code where I hit on this issue is cross-platform Java code that also runs on linux. Unfortunately, this means I can't use kqueue with a kevent's data field to figure out how much I can read without blocking - this data is lost in the abstraction over epoll/kqueue that I use. This means a solution of using a blocking fd à la this answer is unacceptable.

Edit: the original code used kqueue to block on the file descriptor in the read end, which performed worse

Edit 2: Linux os.read() doesn't throw a BlockingIOError before the other side of the pipe is connected despite the docs stating that it should (the call succeeds (returns 0) but sets errno to EAGAIN). Updated the code to be friendly to linux behavior too.

Edit 3: The code for macOS was originally:

import select, os
# mkfifo some_fifo before starting python
fd = os.open('some_fifo',os.O_RDONLY | os.O_NONBLOCK)
kq = select.kqueue()
ke = select.kevent(fd)
while True:
    try:
        read = len(os.read(fd, 8192)) # read up to 8kb (FIFO buffer size in mac os)
    except BlockingIOError:
        evts = kq.control([ke], 1, 10) # 10-second timeout, wait for 1 event
        print(evts)

This performs as poorly as the version with sleeps, but sleeping makes sure the issue isn't with the blocking mechanism, and is cross-platform.

“I have a reader and writer on a FIFO, where the reader must not block indefinitely.” Yet you've written a loop that does nothing but call `read` and `sleep`. Perhaps you could edit your question to explain why you “must not block indefinitely”, and why you can't use `select` or `poll` to check for/wait for more data on the FIFO. — rob mayoff, Feb 14 '18 at 14:51
The original version of the post used kqueue for blocking, but that's not the issue - performance was the same. — Roee Shenberg, Feb 14 '18 at 15:22
Edited to re-add the kqueue version, it works as poorly. The issue is that a 4mb write should be almost instant, no matter how I wait on it. 0.5 seconds for a sleep is ages. — Roee Shenberg, Feb 14 '18 at 15:29
As to why I can't block indefinitely: the actual code I encountered this issue in is a complicated system, and if I don't get a response in 10 seconds it's considered an error and I should move on to do other things. — Roee Shenberg, Feb 14 '18 at 15:30
And finally, select/poll will tell me that I can read from the fd, but not how much. It must be a non-blocking fd so that I don't block on reading if I ask for too much data, so using select/poll don't let my use a blocking fd without risk of blocking. — Roee Shenberg, Feb 14 '18 at 15:35

Poor performance on large writes on O_NONBLOCK fifo in mac os

0 Answers0