Python subprocess get stuck at communicate() call

Question

Context:

I am using python 2.7.5.

I need to run a subprocess from a python script, wait for its termination and get the output.

The subprocess is run around 1000 times.

In order to run my subprocess, I have defined a function:

def run(cmd):
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    (stdout, stderr) = p.communicate()
    return (p.returncode, stdout, stderr)

The subprocess to be executed is a bash script and is passed as the cmd parameter of the run() function.
The command and its arguments are given through a list (as expected by Popen()).

Issue:

In the past, it has always worked without any error.
But recently, the python script get always stuck on a subprocess call after having successfully executed a lot of calls. The subprocess in question is not executed at all (the bash script is not even started) and the python script blocks.

After stopping the execution with Ctrl+C, I get the point where it was stuck:

  [...]

  File "import_debug.py", line 20, in run
    (stdout, stderr) = p.communicate()
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
    ready = poller.poll()
KeyboardInterrupt

I don't understand why I have this issue nor how to solve it.

I have found this SO thread that seems to tackle the same issue or something equivalent (since the output after the keyboard interruption is the same) but there is no answer.

Question: What is happening here ? What am I missing ? How to solve this issue ?

EDIT:

The call is under the form:

(code, out, err) = run(["/path/to/bash_script.sh", "arg1", "arg2", "arg3"])

print out
if code:
    print "Failed: " + str(err)

The bash script is doing some basic processing with the data (unzip archives and do something with the extracted data).
When the error occurs, none of the bash script instructions are executed.

_{I cannot provide the exact command, arguments and contents for company privacy concerns.}

Can you share the bash script that you're trying to execute? and also what you're passing as `cmd`? — prithajnath, Jan 02 '20 at 15:27
@prithajnath I have edited the question to provide the requested information (I cannot give the exact code and data though since it is related to my work and under company privacy). — Fareanor, Jan 02 '20 at 15:47
Personally, i would check if the bash script isn't stuck, maybe unzip blocks because a file/directory exists and asks for input **do you want to overwrite ?** or so. — Maurice Meyer, Jan 02 '20 at 16:12
@MauriceMeyer I already have tested it. But if I run the script on the stuck sample only, it works fine, I have the issue at full processing (~1000 calls). — Fareanor, Jan 03 '20 at 07:58
@Fereaor: Maybe this: https://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy. You could try to redirect stdout/stderr into a file instead of buffering it. — Maurice Meyer, Jan 03 '20 at 08:09
@MauriceMeyer Actually, my program output is very small (1 line if there's no error). But I will try your solution for my next run. Thanks for your help. I'll let you know when it's done. — Fareanor, Jan 03 '20 at 08:14
I have found the issue, the data that was given to me (to process them) was "corrupted". That was unzip that was asking for a replacement, a renaming or a cancellation (the zip file contained another zip with the same name... a non-sense). As my script writes nothing to stdin, it was waiting forever for a response. — Fareanor, Jan 03 '20 at 11:49

lenik · Answer 1 · 2020-01-03T15:29:09.793

3

The author of the original thread you're referring to says: "If I set stderr=None instead of stderr=subprocess.PIPE I never see this issue." -- I'd recommend to do exactly that and get your script working.

Added after reading the comment section:

There are a few useful options, you may want or not to use:

-f  freshen existing files, create none
-n  never overwrite existing files         
-o  overwrite files WITHOUT prompting

edited Jan 03 '20 at 15:29

answered Jan 02 '20 at 15:43

lenik

23,228
4
34
43

I cannot do this because I need to be able to report stderr too if an error occurs. – Fareanor Jan 02 '20 at 15:45
@Fareanor you may redirect `stderr` to `stdout` in your bash script – lenik Jan 03 '20 at 08:45
1

I have found the issue, the data that was given to me (to process them) was "corrupted". That was unzip that was asking for a replacement, a renaming or a cancellation. As my script writes nothing to stdin, it was waiting forever for a response. – Fareanor Jan 03 '20 at 11:49
And I don't want neither to get rid of stderr nor to redirect it in stdout. I want to handle them separately. But thanks anyway for your suggestions. – Fareanor Jan 03 '20 at 11:50

Python subprocess get stuck at communicate() call

1 Answers1