64

I'm writing a Python script that needs to write some data to a temporary file, then create a subprocess running a C++ program that will read the temporary file. I'm trying to use NamedTemporaryFile for this, but according to the docs,

Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later).

And indeed, on Windows if I flush the temporary file after writing, but don't close it until I want it to go away, the subprocess isn't able to open it for reading.

I'm working around this by creating the file with delete=False, closing it before spawning the subprocess, and then manually deleting it once I'm done:

fileTemp = tempfile.NamedTemporaryFile(delete = False)
try:
    fileTemp.write(someStuff)
    fileTemp.close()
    # ...run the subprocess and wait for it to complete...
finally:
    os.remove(fileTemp.name)

This seems inelegant. Is there a better way to do this? Perhaps a way to open up the permissions on the temporary file so the subprocess can get at it?

Nathan Reed
  • 3,583
  • 1
  • 26
  • 33
  • Have you considered piping the processes as a possibility instead? – Jon Clements Mar 02 '13 at 09:46
  • @JonClements Yes, piping is possible too, but it would require a change to the interface of the subprocess, which I'm hoping to avoid. – Nathan Reed Mar 02 '13 at 21:48
  • 2
    Related: Python issue 14243 - [tempfile.NamedTemporaryFile not particularly useful on Windows](http://bugs.python.org/issue14243) – Piotr Dobrogost Mar 05 '13 at 21:10
  • See also a solution for Linux: [Can I set the umask for tempfile.NamedTemporaryFile in python?](https://stackoverflow.com/a/44130605/507544) – nealmcb Nov 14 '17 at 05:35

6 Answers6

27

According to Richard Oudkerk

(...) the only reason that trying to reopen a NamedTemporaryFile fails on Windows is because when we reopen we need to use O_TEMPORARY.

and he gives an example of how to do this in Python 3.3+

import os, tempfile

DATA = b"hello bob"

def temp_opener(name, flag, mode=0o777):
    return os.open(name, flag | os.O_TEMPORARY, mode)

with tempfile.NamedTemporaryFile() as f:
    f.write(DATA)
    f.flush()
    with open(f.name, "rb", opener=temp_opener) as f:
        assert f.read() == DATA

assert not os.path.exists(f.name)

Because there's no opener parameter in the built-in open() in Python 2.x, we have to combine lower level os.open() and os.fdopen() functions to achieve the same effect:

import subprocess
import tempfile

DATA = b"hello bob"

with tempfile.NamedTemporaryFile() as f:
    f.write(DATA)
    f.flush()

    subprocess_code = \
    """import os
       f = os.fdopen(os.open(r'{FILENAME}', os.O_RDWR | os.O_BINARY | os.O_TEMPORARY), 'rb')
       assert f.read() == b'{DATA}'
    """.replace('\n', ';').format(FILENAME=f.name, DATA=DATA)

    subprocess.check_output(['python', '-c', subprocess_code]) == DATA
Piotr Dobrogost
  • 41,292
  • 40
  • 236
  • 366
  • Interesting...I'm in Python 2.x but I'll have to look into this further. – Nathan Reed Mar 06 '13 at 02:22
  • @NathanReed It's also possible in Python 2.x - see my update. – Piotr Dobrogost Mar 13 '13 at 20:10
  • 10
    Note that this only works if you control the code in the subprocess. Calling a system utility like `tar` won't trivially work. Still, neat find! – Dave Dopson Nov 20 '15 at 21:20
  • It's interesting because the documentation says that os.O_TEMPORARY is only available in Linux: http://bit.ly/1MmDUxz Maybe they have added support for that value in windows and the python implementation of NamedTempraryFile is not updated for windows ? – yucer Mar 16 '16 at 14:59
  • 1
    @yucer Look carefully; label for each group of constants in the page you showed is under this group not above. – Piotr Dobrogost Mar 16 '16 at 15:30
  • @PiotrDobrogost Thanks ! I find that rare. Is there some kind of PEP for the documentation ? – yucer Mar 16 '16 at 15:45
  • @yucer I've never heard about one and I think there is no such PEP. – Piotr Dobrogost Mar 16 '16 at 16:03
  • 4
    This is the best solution if you control the code of the subprocess. If not, I don't see more elegant solution than the one from original question. – Marcin Raczyński Apr 09 '18 at 08:33
  • You can simplify this. No need to do the `fdopen()` dance. Have a look at the documentation for `fopen()` modes: https://msdn.microsoft.com/en-us/library/yeby3zcb.aspx the mode 'D' does what is needed. So just replace the usual mode `'rb'` with `'rbD'` in the open call and it works. Sadly `io.open(fn, "rbD")` throws a `ValueError` for this mode. – schlenk Nov 15 '18 at 18:11
27

Since nobody else appears to be interested in leaving this information out in the open...

tempfile does expose a function, mkdtemp(), which can trivialize this problem:

try:
    temp_dir = mkdtemp()
    temp_file = make_a_file_in_a_dir(temp_dir)
    do_your_subprocess_stuff(temp_file)
    remove_your_temp_file(temp_file)
finally:
    os.rmdir(temp_dir)

I leave the implementation of the intermediate functions up to the reader, as one might wish to do things like use mkstemp() to tighten up the security of the temporary file itself, or overwrite the file in-place before removing it. I don't particularly know what security restrictions one might have that are not easily planned for by perusing the source of tempfile.

Anyway, yes, using NamedTemporaryFile on Windows might be inelegant, and my solution here might also be inelegant, but you've already decided that Windows support is more important than elegant code, so you might as well go ahead and do something readable.

Corbin
  • 1,530
  • 1
  • 9
  • 19
  • 3
    The cleanup code can be simplified with `shutil.rmtree(temp_dir)` – Dave Dopson Nov 20 '15 at 21:21
  • I could be mistaken, but this looks like the best answer. – Russia Must Remove Putin Feb 04 '16 at 21:33
  • 4
    Caution! **This does not answer original question at all.** Not only this is no better than what OP is already doing but it's much worse leaving fragile aspects of creation of temporary file to the user –`make_a_file_in_a_dir(temp_dir)`. Please see my answer for a much better way to handle this. (I do not understand why I hadn't written this comment four years earlier when I first read this...). – Piotr Dobrogost Mar 07 '17 at 08:57
  • 2
    Downvote. Although this answer proposes solution to original problem (opening temporary file in a subprocess) this is NOT more elegant answer than the answer included in the original question because this answer uses a lower level function and ignores general temporary file problems (ex. security). – Marcin Raczyński Apr 09 '18 at 08:32
  • 1
    `local variable 'temp_dir' referenced before assignment`. I guess the actual creation has to go out of the "try", which makes it hard to debug it? – Hugues Fontenelle Oct 08 '18 at 14:46
  • 1
    I have just tried this and my version didn't work. The create temporary file was: `cert_file = tempfile.NamedTemporaryFile(dir=temp_dir)` – hum3 Mar 05 '19 at 11:46
  • 1
    This has the virtue of working for subprocesses you have no control over. – AdamC Jul 08 '19 at 15:48
13

You can always go low-level, though am not sure if it's clean enough for you:

fd, filename = tempfile.mkstemp()
try:
    os.write(fd, someStuff)
    os.close(fd)
    # ...run the subprocess and wait for it to complete...
finally:
    os.remove(filename)
tshepang
  • 12,111
  • 21
  • 91
  • 136
11

At least if you open a temporary file using existing Python libraries, accessing it from multiple processes is not possible in case of Windows. According to MSDN you can specify a 3rd parameter (dwSharedMode) shared mode flag FILE_SHARE_READ to CreateFile() function which:

Enables subsequent open operations on a file or device to request read access. Otherwise, other processes cannot open the file or device if they request read access. If this flag is not specified, but the file or device has been opened for read access, the function fails.

So, you can write a Windows specific C routine to create a custom temporary file opener function, call it from Python and then you can make your sub-process access the file without any error. But I think you should stick with your existing approach as it is the most portable version and will work on any system and thus is the most elegant implementation.

  • Discussion on Linux and windows file locking can be found here.

EDIT: Turns out it is possible to open & read the temporary file from multiple processes in Windows too. See Piotr Dobrogost's answer.

Community
  • 1
  • 1
Nilanjan Basu
  • 828
  • 9
  • 20
  • Thanks. I wish the Python API would expose those mode flags but I guess I can see why they don't. Oh well. – Nathan Reed Mar 02 '13 at 21:47
  • *At least if you open a temporary file using existing Python libraries, accessing it from multiple processes is not possible in case of Windows.* That's not true. See http://bugs.python.org/issue14243#msg164504 and my [answer](http://stackoverflow.com/a/15235559/95735). – Piotr Dobrogost Mar 13 '13 at 20:40
0

Using mkstemp() instead with os.fdopen() in a with statement avoids having to call close():

fd, path = tempfile.mkstemp()
try:
    with os.fdopen(fd, 'wb') as fileTemp:
        fileTemp.write(someStuff)
    # ...run the subprocess and wait for it to complete...
finally:
    os.remove(path)
Jon-Eric
  • 16,977
  • 9
  • 65
  • 97
0

I know this is a really old post, but I think it's relevant today given that the API is changing and functions like mktemp and mkstemp are being replaced by functions like TemporaryFile() and TemporaryDirectory(). I just wanted to demonstrate in the following sample how to make sure that a temp directory is still available downstream:

Instead of coding:

tmpdirname = tempfile.TemporaryDirectory()

and using tmpdirname throughout your code, you should trying to use your code in a with statement block to insure that it is available for your code calls... like this:

with tempfile.TemporaryDirectory() as tmpdirname:
    [do dependent code nested so it's part of the with statement]

If you reference it outside of the with then it's likely that it won't be visible anymore.

  • The original question is about a temporary file. Not a temporary directory. Do you have an example for temporary file? – howdoicode Sep 08 '22 at 11:22