1

I seem to be having an issue with Python when I run a script that creates a large number of sub processes. The sub process creation code looks similar to:

Code:

def execute(cmd, stdout=None, stderr=subprocess.STDOUT, cwd=None):
    proc = subprocess.Popen(cmd, shell=True, stdout=stdout, stderr=stderr, cwd=cwd)
    atexit.register(lambda: __kill_proc(proc))
    return proc

The error message I am receiving is:

OSError: [Errno 24] Too many open files

Once this error occurs, I am unable to create any further sub processes until kill the script and start it again. I am wondering if the following line could be responsible.

atexit.register(lambda: __kill_proc(proc))

Could it be that this line creates a reference to the sub process, resulting in a "file" remaining open until the script exits?

torbinsky
  • 1,450
  • 10
  • 17
  • What do you do with `proc`? Do you call `communicate()` or `wait()` or `terminate()/kill()`? If not, then yes probably that's what's going on. – Bakuriu Sep 21 '12 at 02:49
  • 1
    Also you could rewrite `__kill_proc` in order to use the process pid instead of the `Popen` instance. In this way you don't keep references around. Try it and see if the behaviour changes. – Bakuriu Sep 21 '12 at 02:55
  • In response to your first comment: Typically communicate()/wait() are called. In response to second comment: Ahh, good suggestion. I'll give that a try and see if it solves the problem. – torbinsky Sep 21 '12 at 03:42

2 Answers2

1

So it seems that the line:

atexit.register(lambda: __kill_proc(proc))

was indeed the culprit. This is probably because of the Popen reference being kept around so the process resources aren't free'd. When I removed that line the error went away. I have now changed the code as @Bakuriu suggested and am using the process' pid value rather than the Popen instance.

torbinsky
  • 1,450
  • 10
  • 17
0

Firstly, run ulimit -a to find out how many the maximum open files are set in your Linux system.

Then edit the system configuration file /etc/security/limits.conf and add those code in the bottom.

* - nofile 204800

Then you can open more sub processes if you want.

LexusLee
  • 21
  • 2