13

When I use multiprocessing.Queue.get I sometimes get an exception due to EINTR.

I know definitely that sometimes this happens for no good reason (I open another pane in a tmux buffr), and in such a case I would want to continue working and retry the operation.

I can imagine that in some other cases The error would be due to a good reason and I should stop running or fix some error.

How can I distinguish the two?

Thanks in advance

Shwouchk
  • 553
  • 2
  • 5
  • 10
  • you should make sure the queue is not empty before `get`: `if not queue.empty(): queue.get()` – Inbar Rose Jan 03 '13 at 09:39
  • I believe that if the queue is empty, it will just block the call. Won't it? In any I don't think this is the cause of the error. – Shwouchk Jan 03 '13 at 10:26

2 Answers2

18

The EINTR error can be returned from many system calls when the application receives a signal while waiting for other input. Typically these signals can be quite benign and already handled by Python, but the underlying system call still ends up being interrupted. When doing C/C++ coding this is one reason why you can't entirely rely on functions like sleep(). The Python libraries sometimes handle this error code internally, but obviously in this case they're not.

You might be interested to read this thread which discusses this problem.

The general approach to EINTR is to simply handle the error and retry the operation again - this should be a safe thing to do with the get() method on the queue. Something like this could be used, passing the queue as a parameter and replacing the use of the get() method on the queue:

import errno

def my_queue_get(queue, block=True, timeout=None):
    while True:
        try:
            return queue.get(block, timeout)
        except IOError, e:
            if e.errno != errno.EINTR:
                raise

# Now replace instances of queue.get() with my_queue_get(queue), with other
# parameters passed as usual.

Typically you shouldn't need to worry about EINTR in a Python program unless you know you're waiting for a particular signal (for example SIGHUP) and you've installed a signal handler which sets a flag and relies on the main body of the code to pick up the flag. In this case, you might need to break out of your loop and check the signal flag if you receive EINTR.

However, if you're not using any signal handling then you should be able to just ignore EINTR and repeat your operation - if Python itself needs to do something with the signal it should have already dealt with it in the signal handler.

Cartroo
  • 4,233
  • 20
  • 22
  • I can't vote up because i have too little reputation... Thanks for the answer! – Shwouchk Jan 12 '13 at 15:11
  • Without too much knowledge it wasnt so obvious what to do, but I replaced result.get() with my_queue_get(result). I believe thats what should be done incase others are wondering how to use the code.. – user-2147482637 Mar 09 '15 at 10:08
  • Yes, it was a little vague - I've updated the answer. – Cartroo Mar 09 '15 at 14:10
8

Old question, modern solution: as of Python 3.5, the wonderful PEP 475 - Retry system calls failing with EINTR has been implemented and solves the problem for you. Here is the abstract:

System call wrappers provided in the standard library should be retried automatically when they fail with EINTR , to relieve application code from the burden of doing so.

By system calls, we mean the functions exposed by the standard C library pertaining to I/O or handling of other system resources.

Basically, the system will catch and retry for you a piece of code that failed with EINTR so you don't have to handle it anymore. If you are targeting an older release, the while True loop still is the way to go. Note however that if you are using Python 3.3 or 3.4, you can catch the dedicated exception InterruptedError instead of catching IOError and checking for EINTR.

Community
  • 1
  • 1
Morwenn
  • 21,684
  • 12
  • 93
  • 152