automatically closing a file in python3?

Question

Am I right in thinking that this will automatically close a file?

def get_file():
    with open("file.csv", "rb") as f:
        yield f

f = get_file()
do_stuff(f)

If not, how do i write a function that returns a file object, whilst making sure that it closes the file after the receiver is done using it?

the code is useless: you get a handle on a closed file. `yield f.read()` would work — Jean-François Fabre, Jul 25 '19 at 13:25
also opening a csv file in binary mode doesn't work anymore in python 3 — Jean-François Fabre, Jul 25 '19 at 13:25
assuming that you are reading data sequentially and you can do your processing in blocks what you could do is to yield blocks from f rather than f. — norok2, Jul 25 '19 at 13:26
why wouldn't binary mode work with simple open? the code is not using csv module — norok2, Jul 25 '19 at 13:29
@norok2: It's not showing that it's using it, but most code *should* be using it; it's not a bad idea to mention this change in behavior (`open` for CSV code in Py2 has to be substantially changed in Py3). — ShadowRanger, Jul 25 '19 at 13:34

Martijn Pieters · Accepted Answer · 2019-07-27T13:16:35.557

This sort of can be made to work, but is not a good way to go about it. get_file() returns is a generator function, and calling it returns a specialised generator iterator object, not the open file object itself, not directly.

It works when you use the next() on the generator to work with the file:

f = get_file()
do_stuff(next(f))

Here next() advances the generator to the yield point and returns whatever was yielded. At that point the context for the with open(...) as f: remains active and the file is not going to be closed.

However, to then close the file you'd have to call next() again and prevent the StopIteration exception from being raised:

next(f, None)  # give `next()` a default to return when the generator exists

That's not really ideal. You want to wrap your function in a @contextlib.contextmanager() decorator, which requires that the decorated function is a generator. You can then must use get_file() as a context manager:

from contextlib import contextmanager

@contextmanager
def get_file():
    with open("file.csv", "rb") as f:
        yield f

with get_file() as f:
    do_stuff(f)

There is not much point in using get_file() like this, because you may as well just use return open("file.csv", "rb") and rely on the file object itself being the context manager. But if you were to add other tasks to the get_file() function that need access to the file or need to know that you closed it, then you may well have a good use-case for a custom context manager.

Note: Rather than calling `next` again, you *could* call `f.close()`; [generator-iterators support `close`](https://docs.python.org/3/reference/expressions.html#generator.close) to explicitly raise `GeneratorExit` in the generator, which bubbles up, allowing `try` and `with` blocks to do their job properly. Converting to a context manager is a better solution, but then, in this case you could just change `get_file`'s body to just `return open("file.csv", "rb")` for the same effect (the caller must still use it with `with` for safety, but the function is simpler). — ShadowRanger, Jul 26 '19 at 16:17
@ShadowRanger: Yes, the file object itself is already a context manager; the assumption is that the `get_file()` function would do more than just open the file. It depends on what 'do more' entails if `generator.close()` is a good alternative, since that exits the normal function flow. — Martijn Pieters, Jul 27 '19 at 13:14

score 0 · Answer 2 · answered Jul 25 '19 at 13:31

This most likely won't work. Using yield in a function makes the function return a generator which will produce the first yielded value as first element, and will run the function until the next yield when another element is needed.

That is not what you expect. I'm afraid that it is impossible to decide when the calling code is done with the file. What might be possible (I don't know Python's memory management well enough) is that the file's operating system handle (file descriptor) is closed when the file is garbage collected. However, this would require that you clear out all direct and indirect references, which could be tricky.

It's much easier to perform the complete file handling within the with open() block, and it's much more readable.

automatically closing a file in python3?

2 Answers2