Old question, but if I could add to @Mehdrad's answer, from a slightly different standpoint...
On Windows, deleting a file often isn't even completely synchronous, and it isn't even a single operation. In that sense it's definitely not atomic.
If you look at tools like process monitor or look at MSFT documentation for writing a filesystem driver, you'll notice that to delete a file on Windows is a multi-step process. First you need a handle to the file. Then you set its disposition to "deleted". This puts the file in a state where it has a "delete pending". The file won't even be removed from view until the last handle to it is closed. When a file is in this state, new attempts to open the file will fail with STATUS_DELETE_PENDING
. This status is more of a runtime thing -- if you pulled the plug or did a reboot the files won't stay in that state.
So, it may or may not be relevant to the way you're using deletes, but it's important to keep in mind that on Windows a delete won't necessary take effect right away and under concurrent access may lock further requests out of getting to the file.