7

I am looking for an atomic version of the following:

import os

def tryMakeFile(filename):
    try:
        with open(filename) as _:
            return False
    except FileNotFoundError:
        with open(filename, mode='a') as _:
            return True

(Please don't comment on stylistic issues here - I know this code is bad in many ways, but it suffices to illustrate my question.)

In other words, I'm looking for a way to check if a file exists, and create it if it doesn't, in Python, in such a way that I know which happened. But done in such a way that there isn't a race condition between multiple processes (in my given example code, two processes could both think they created the file, if the second process ran while the first was suspended between the first and second open calls).

Or, to put it another way, I am looking for a Python equivalent of Java's Files.createFile call.

Edit: note that when I say "Python" I mean "portable Python". Saying "use this library* (*this library is only available on Windows, or not on Windows, or only on the second Tuesday after a blue moon)" isn't what I'm looking for. I'm looking for something that is explicitly atomic, part of the standard library and/or builtins, and it's available on common platforms.

TLW
  • 1,373
  • 9
  • 22
  • what about https://pypi.python.org/pypi/lockfile ? – Stephane Martin Oct 19 '15 at 20:51
  • That library relies on `os.mkdir` being atomic, which I don't see as being documented anywhere. – TLW Oct 19 '15 at 21:48
  • fcntl.lockf ? fcntl.flock ? – Stephane Martin Oct 19 '15 at 22:02
  • Not portable, as far as I know. I'd love to be wrong, though. – TLW Oct 19 '15 at 22:05
  • flock is posix, you should have it on pretty much everything but windows. for Windows API, LockFile and LockFileEx syscalls i think (haven't checked). – Stephane Martin Oct 19 '15 at 22:11
  • 1
    So introduce a dependency to win32api module, require two different implementations for different systems, and then still break portability for systems not covered by those implementations? Bleh. I'm looking for something that's portable as in "if something implements Python and the standard library correctly it'll work". That's not it. – TLW Oct 19 '15 at 22:27

3 Answers3

17

You can use os.open with os.O_CREAT | os.O_EXCL flags which will fail if the file exists, they are according to the docs available on Unix and Windows but I am not sure if atomic file creation exists on windows or not:

os.open("filename", os.O_CREAT | os.O_EXCL)

From the linux open man page:

O_EXCL If O_CREAT and O_EXCL are set, open() shall fail if the file exists. The check for the existence of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing open() naming the same filename in the same directory with O_EXCL and O_CREAT set. If O_EXCL and O_CREAT are set, and path names a symbolic link, open() shall fail and set errno to [EEXIST], regardless of the contents of the symbolic link. If O_EXCL is set and O_CREAT is not set, the result is undefined.

Not sure what you want to do if the file exists but you just need to catch a FileExistsError when the file does already exist:

import os

def try_make_file(filename):
    try:
        os.open(filename,  os.O_CREAT | os.O_EXCL)
        return True
    except FileExistsError:
        return False
Padraic Cunningham
  • 176,452
  • 29
  • 245
  • 321
  • 1
    When it says other threads, is that only within the same process, or does that extend to other processes as well? – TLW Oct 19 '15 at 21:29
  • 1
    It includes other processes in linux, I cannot speak for windows as I never use it, using O_EXCL is used to avoid race conditions in lock files etc... – Padraic Cunningham Oct 19 '15 at 21:39
12

If you have Python 3.3 or better, you could use the 'x' mode with open():

'x' open for exclusive creation, failing if the file already exists

def tryMakeFile(filename):
    try:
        with open(filename, "x") as _:
            return False
    except FileExistsError:
        return True
Eugene Yarmash
  • 142,882
  • 41
  • 325
  • 378
  • 1
    That mentions nothing about atomicity, however. – TLW Oct 19 '15 at 21:38
  • 2
    @TLW: It seems to be a shortcoming of the documentation. The 'exclusive creation' semantics does imply atomicity. See also http://stackoverflow.com/a/29295749/244297 – Eugene Yarmash Oct 19 '15 at 22:23
  • 3
    That link does not explain anything beyond a restating of the documentation. – TLW Oct 19 '15 at 22:35
  • @TLW - it's _one function call_: `open("filename",'x')`. If it's got _internal_ non-atomicities that will cause it to fail, that's an internal race-condition which would be documented. – JamesTheAwesomeDude Aug 14 '20 at 16:55
  • @JamesTheAwesomeDude - _many_ Python standard library functions are nonatomic under the hood, with no documentation. (E.g. `os.listdir` is not atomic, as it uses potentially-many calls to readdir under the hood. https://github.com/python/cpython/blob/main/Modules/posixmodule.c#4089 .) It is my understanding that Python typically documents the _atomic_ functions, not the _non_atomic ones. – TLW Oct 02 '22 at 00:01
  • @TLW Yes, and since the function is _advertised_ as "exclusive creation", if it _weren't_ actually atomic that'd be a problem. It's my understanding that exclusive creation is an atomic primitive that you actually get guarantees from; is this incorrect? – JamesTheAwesomeDude Oct 03 '22 at 15:23
5

There's another variation of this, using pathlib.Path:

from pathlib import Path

def try_make_file(filename):
    try:
        Path(filename).touch(exist_ok=False)
        return True
    except FileExistsError:
        return False

It is not explicitly documented, but in the source code we can see that this implies the os.O_EXCL flag:

if not exist_ok:
    flags |= os.O_EXCL

See the function definition in the pathlib source code.

As such this carries the same properties as the other solutions (namely that it is unclear if this works on Windows).

href_
  • 1,509
  • 16
  • 15