Python: Check if named pipe has data

Question

I have a Python3 process on my Unix system always running, and I want to be able to randomly send data to it through a named pipe from other processes that only run occasionally. If the named pipe doesn't have data, I want my process to continue doing other things, so I need to check whether it has data without blocking.

I can't figure out how to check without opening it, but opening blocks unless I set the non-blocking flag. And if I set the flag, it crashes if I happen to write to the pipe before or during the read.

This is the best I've managed to do:

import os

fifo = "pipe_test.fifo"
done = False
fd = os.open(fifo, os.O_RDONLY | os.O_NONBLOCK)
while not done:
    try:
        s = os.read(fd, 1024) # buffer size may need tweaking
        print(s)
        done = True
    except BlockingIOError as e:
        pass
os.close(fd)

If there's no data in the pipe, I get b"", and it exits. If there's data in the pipe, it gets an exception once, retries, then gets the data. Seems like I'm doing something wrong and might run into weird race conditions. Is there a nicer way to do this?

Could you use a second thread that waits for the blocking pipe? — Jasper, May 25 '16 at 20:21
@Jasper I could do that. That's not ideal either, but I admit it's better. Also, I can use Python's file library instead of the low-level `os` stuff that way, which allows me to read up until newlines and such. Thanks, I'm going to change my code to that unless there's a way to check without another thread. — sudo, May 25 '16 at 20:34

score 4 · Accepted Answer · answered May 25 '16 at 20:33

4

I would not use a named-pipe if you can change the clients' code, but UNIX domain sockets instead, because they support datagrams:

import errno, fcntl, os, socket

Server:

# bind socket
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.bind('pipe_test.fifo')
# set socket non-blocking
fcntl.fcntl(sock.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

# get a datagram
try:
    datagram = sock.recv(1024)
except (OSError, socket.error) as ex:
    if ex.errno not in (errno.EINTR, errno.EAGAIN):
        raise
else:
    print('Datagram: %r' % datagram)

Client:

sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.sendto('Hello!', 'pipe_test.fifo')

But you might want to look into multithreading instead of using non-blocking sockets.

answered May 25 '16 at 20:33

Kijewski

25,517
12
101
143

I didn't know about datagrams. That's useful. But it looks like you're throwing and catching errors similarly to how I am. Is that safe? – sudo May 25 '16 at 21:40
@sudo, it is. If there is no datagram waiting to be received, then EAGAIN is thrown. If a signal interrupted the receiving, then EINTR is thrown. The latter case shouldn't be impossible for non-blocking sockets, but it does not hurt to be safe. Possible errors are: a too little read buffer (but you know your data), and that clients can't send data, because you read the too slowly. The buffer of a socket is 200kB or 512 packets by default, http://stackoverflow.com/a/22007520/416224. – Kijewski May 25 '16 at 22:03

sudo · Answer 2 · 2016-05-25T21:50:20.093

This isn't really the answer, but in case it's useful to anyone, here's how I'm doing it with another thread.

class QueryThread(threading.Thread):

    def __init__(self, args=(), kwargs=None):
        threading.Thread.__init__(self, args=(), kwargs=None)
        self.daemon = True
        self.buf = []
        if not general.f_exists("pipe"):
            os.mkfifo("pipe")

    def run(self):
        f = open("pipe")
        while True:
            try:
                query = next(f).replace("\n", "")
                if query != "":
                    self.buf.append(query)
                    print("Read in new query from pipe: {}, buf = {}".format(query, self.buf))
            except StopIteration: # not a pipe error, just means no data is left, so time to re-open
                f.close()
                f = open("pipe")
        f.close()

    def get_query(self):
        if len(self.buf) == 0: return ""
        query = self.buf[0]
        self.buf.__delitem__(0)
        return query

It keeps newline-deliminated messages in a buffer. You can call the get_query method from another thread and get the last message that was received.

Python: Check if named pipe has data

2 Answers2

Linked