How to preserve ownership and permissions when doing an atomic file replace?

Question

So, the normal POSIX way to safely, atomically replace the contents of a file is:

• fopen(3) a temporary file on the same volume
• fwrite(3) the new contents to the temporary file
• fflush(3)/fsync(2) to ensure the contents are written to disk
• fclose(3) the temporary file
• rename(2) the temporary file to replace the target file

However, on my Linux system (Ubuntu 16.04 LTS), one consequence of this process is that the ownership and permissions of the target file change to the ownership and permissions of the temporary file, which default to uid/gid and current umask.

I thought I would add code to stat(2) the target file before overwriting, and fchown(2)/fchmod(2) the temporary file before calling rename, but that can fail due to EPERM.

Is the only solution to ensure that the uid/gid of the file matches the current user and group of the process overwriting the file? Is there a safe way to fall back in this case, or do we necessarily lose the atomic guarantee?

Answer 1

Is the only solution to ensure that the uid/gid of the file matches the current user and group of the process overwriting the file?

No.

In Linux, a process with the CAP_LEASE capability can obtain an exclusive lease on the file, which blocks other processes from opening the file for up to /proc/sys/fs/lease-break-time seconds. This means that technically, you can take the exclusive lease, replace the file contents, and release the lease, to modify the file atomically (from the perspective of other processes).

Also, a process with the CAP_CHOWN capability can change the file ownership (user and group) arbitrarily.

Is there a safe way to [handle the case where the uid or gid does not match the current process], or do we necessarily lose the atomic guarantee?

Considering that in general, files may have ACLs and xattrs, it might be useful to create a helper program, that clones the ownership including ACLs, and extended attributes, from an existing file to a new file in the same directory (perhaps with a fixed name pattern, say .new-################, where # indicate random alphanumeric characters), if the real user (getuid(), getgid(), getgroups()) is allowed to modify the original file. This helper program would have at least the CAP_CHOWN capability, and would have to consider the various security aspects (especially the ways it could be exploited). (However, if the caller can overwrite the contents, and create new files in the target directory -- the caller must have write access to the target directory, so that they can do the rename/hardlink replacement --, creating a clone file on their behalf with empty contents ought to be safe. I would personally exclude target files owned by root user or group, though.)

Essentially, the helper program would behave much like the mktemp command, except it would take the path to the existing target file as a parameter. It would then be relatively straightforward to wrap it into a library function, using e.g. fork()/exec() and pipes or sockets.

I personally avoid this problem by using group-based access controls: dedicated (local) group for each set. The file owner field is basically just an informational field then, indicating the user that last recreated (or was in charge of) said file, with access control entirely based on the group. This means that changing the mode and the group id to match the original file suffices. (Copying ACLs would be even better, though.) If the user is a member of the target group, they can do the fchown() to change the group of any file they own, as well as the fchmod() to set the mode, too.

Answer 2

I am by no means an expert in this area, but I don't think it's possible. This answer seems to back this up. There has to be a compromise.

Here are some possible solutions. Every one has advantages and disadvantages and weighted and chosen depending on the use case and scenario.

• Use atomic rename.

  Advantage: atomic operation

  Disadvantage: possible to not keep owner/permissions

• Create a backup. Write file in place

  This is what some text editor do.

  Advantage: will keep owner/permissions

  Disadvantage: no atomicity. Can corrupt file. Other application might get a "draft" version of the file.

• Set up permissions to the folder such that creating a new file is possible with the original owner & attributes.

  Advantages: atomicity & owner/permissions are kept

  Disadvantages: Can be used only in certain specific scenarios (knowledge at the time of creation of the files that would be edited, the security model must allow and permit this). Can decrease security.

• Create a daemon/service responsible for editing the files. This process would have the necessary permissions to create files with the respective owner & permissions. It would accept requests to edit files.

  Advantages: atomicity & owner/permissions are kept. Higher and granular control to what and how can be edited.

  Disadvantages. Possible in only specific scenarios. More complex to implement. Might require deployment and installation. Adding an attack surface. Adding another source of possible (security) bugs. Possible performance impact due to the added intermediate layer.

Answer 3

• Do you have to worry about the file that's named being a symlink to a file somewhere else in the file system?
• Do you have to worry about the file that's named being one of multiple links to an inode (st_nlink > 1).
• Do you need to worry about extended attributes?
• Do you need to worry about ACLs?
• Does the user ID and group IDs of the current process permit the process to write in the directory where the file is stored?
• Is there enough disk space available for both the old and the new files on the same file system?

Each of these issues complicates the operation.

Symlinks are relatively easy to deal with; you simply need to establish the realpath() to the actual file and do file creation operations in the directory containing the real path to the file. From here on, they're a non-issue.

In the simplest case, where the user (process) running the operation owns the file and the directory where the file is stored, can set the group on the file, the file has no hard links, ACLs or extended attributes, and there's enough space available, then you can get atomic operation with more or less the sequence outlined in the question — you'd do group and permission setting before executing the atomic rename() operation.

There is an outside risk of TOCTOU — time of check, time of use — problems with file attributes. If a link is added between the time when it is determined that there are no links and the rename operation, then the link is broken. If the owner or group or permissions on the file change between the time when they're checked and set on the new file, then the changes are lost. You could reduce the risk of that by breaking atomicity but renaming the old file to a temporary name, renaming the new file to the original name, and rechecking the attributes on the renamed old file before deleting it. That is probably an unnecessary complication for most people, most of the time.

If the target file has multiple hard links to it and those links must be preserved, or if the file has ACLs or extended attributes and you don't wish to work out how to copy those to the new file, then you might consider something along the lines of:

1. write the output to a named temporary file in the same directory as the target file;
2. copy the old (target) file to another named temporary file in the same directory as the target;
3. if anything goes wrong during steps 1 or 2, abandon the operation with no damage done;
4. ignoring signals as much as possible, copy the new file over the old file;
5. if anything goes wrong during step 4, you can recover from the extra backup made in step 2;
6. if anything goes wrong in step 5, report the file names (new file, backup of original file, broken file) for the user to clean up;
7. clean up the temporary output file and the backup file.

Clearly, this loses all pretense at atomicity, but it does preserve links, owner, group, permissions, ACLS, extended attributes. It also requires more space — if the file doesn't change size significantly, it requires 3 times the space of the original file (formally, it needs size(old) + size(new) + max(size(old), size(new)) blocks). In its favour is that it is recoverable even if something goes wrong during the final copy — even a stray SIGKILL — as long as the temporary files have known names (the names can be determined).

Automatic recovery from SIGKILL probably isn't feasible. A SIGSTOP signal could be problematic too; a lot could happen while the process is stopped.

I hope it goes without saying that errors must be detected and handled carefully with all the system calls used.

If there isn't enough space on the target file system for all the copies of the files, or if the process cannot create files in the target directory (even though it can modify the original file), you have to consider what the alternatives are. Can you identify another file system with enough space? If there isn't enough space anywhere for both the old and the new file, you clearly have major issues — irresolvable ones for anything approaching atomicity.

The answer by Nominal Animal mentions Linux capabilities. Since the question is tagged POSIX and not Linux, it isn't clear whether those are applicable to you. However, if they can be used, then CAP_LEASE sounds useful.

• How crucial is atomicity vs accuracy?
• How crucial is POSIX compliance vs working on Linux (or any other specific POSIX implementation)?