with open inside try - except block, too many files open?

Question

Quite simply, I am cycling through all sub folders in a specific location, and collecting a few numbers from three different files.

def GrepData():
import glob as glob
import os as os

os.chdir('RUNS')
RUNSDir = os.getcwd()
Directories = glob.glob('*.*')
ObjVal = []
ParVal = []
AADVal = []

for dir in Directories:
    os.chdir(dir)
    (X,Y) = dir.split(sep='+')
    AADPath = glob.glob('Aad.out')
    ObjPath = glob.glob('fobj.out')
    ParPath = glob.glob('Par.out')

    try:
        with open(os.path.join(os.getcwd(),ObjPath[0])) as ObjFile:
            for line in ObjFile:
                ObjVal.append(list([X,Y,line.split()[0]]))
        ObjFile.close()
    except(IndexError):
        ObjFile.close()

    try:
        with open(os.path.join(os.getcwd(),ParPath[0])) as ParFile:
            for line in ParFile:
                ParVal.append(list([X,Y,line.split()[0]]))
        ParFile.close()
    except(IndexError):
        ParFile.close()
    try:
        with open(os.path.join(os.getcwd(),AADPath[0])) as AADFile:
            for line in AADFile:
                AADVal.append(list([X,Y,line.split()[0]]))
        AADFile.close()
    except(IndexError):
        AADFile.close()
    os.chdir(RUNSDir)

Each file open command is placed in a try - except block, as in a few cases the file that is opened will be empty, and thus appending the line.split() will lead to an index error since the list is empty.

However when running this script i get the following error: "OSError: [Errno 24] Too Many open files"

I was under the impression that the idea of the "with open..." statement was that it took care of closing the file after use? Clearly that is not happening.

So what I am asking for is two things:

The answer to: "Is my understanding of with open correct?"
How can I correct whatever error is inducing this problem?

(And yes i know the code is not exactly elegant. The whole try - except ought to be a single object that is reused - but I will fix that after figuring out this error)

Curious, since you know the `with` statement closes the file why do you in every case explicitly close it yourself afterwards? — Daniel Roseman, Feb 20 '13 at 17:38
That was my thought as well. Suggestion: remove the `close` calls and try running the code again. If it continues to show the same error, as I suspect it should, then update the code in your question accordingly (i.e. remove the `close()`s). Otherwise, edit your question to say that removing the `close` calls fixes the issue, and perhaps someone will be able to explain why. — David Z, Feb 20 '13 at 17:41
Well - since I got the too many files open error it obviously was not. So I added that (used to be pass) in the hope that the error was due to the exception resulting in the file staying open. — Daniel, Feb 20 '13 at 17:44
It's not one of the `open` here that leaves files not closed, as they are all in `with` statements. I don't see how it is possible... Can you try to reduce it to something that we can run too? For example with instructions like "create 1000 directories with these three files, and run the script like that". — Armin Rigo, Feb 20 '13 at 17:47

score 1 · Accepted Answer · answered Feb 20 '13 at 17:42

Try moving your try-except inside the with like so:

with open(os.path.join(os.getcwd(),ObjPath[0])) as ObjFile:
    for line in ObjFile:
        try:
            ObjVal.append(list([X,Y,line.split()[0]]))
        except(IndexError): 
           pass

Notes: there is no need to close your file manually, this is what with is for. Also, there is no need to use as os in your imports if you are using the same name.

score 0 · Answer 2 · answered Feb 20 '13 at 17:49

"Too many open files" has nothing to do with writing semantically incorrect python code, and you are using with correctly. The key is the part of your error that says "OSError," which refers to the underlying operating system.

When you call open(), the python interpreter will execute a system call. The details of the system call vary a bit by which OS you are using, but on linux this call is open(2). The operating system kernel will handle the system call. While the file is open, it has an entry in the system file table and takes up OS resources -- this means effectively it is "taking up space" whilst it is open. As such the OS has a limit to the number of files that can be opened at any one time.

Your problem is that while you call open(), you don't call close() quickly enough. In the event that your directory structure requires you to have many thousands files open at once that might approach this cap, it can be temporarily changed (at least on linux, I'm less familiar with other OSes so I don't want to go into too many details about how to do this across platforms).

with open inside try - except block, too many files open?

2 Answers2