is there a way to synchronize write to a file among different processes (not threads)

Question

My project requires being run on several different physical machines, which have shared file system among them. One problem arising out of this is how to synchronize write to a common single file. With threads, that can be easily achieved with locks, however my program consists of processes distributed on different machines, which I have no idea how to synchronize. In theory, any way to check whether a file is being opened right now or any lock-like solutions will do, but I just cannot crack out this by myself. A python way would be particularly appreciated.

Is there a way to .. alter this design? Also, by shared, do you mean SMB or other? — user2246674, Jun 29 '13 at 02:31
Why is it a bad design? In fact, if I were able to implement the synchronization, that would be a huge speed up for my project — Daniel, Jun 29 '13 at 02:34
sorry I don't know much about SMB, it seems like a distributed networked file system. pretty much the way Andrew file system is — Daniel, Jun 29 '13 at 02:40
I didn't say it was a "bad" design. I merely made a suggestion to reflect upon it: is this the most suitable (per some metrics or requirements) approach? As for the filesystem itself; different distributed filesystems have [different "quirks"](http://oilq.org/fr/node/13344) (talks about flock) .. in any case, see [this thread](http://mail.python.org/pipermail/python-dev/2007-October/074992.html). — user2246674, Jun 29 '13 at 02:43
great resource! I was also looking into sth as fcntl.lockf, but there's some swing on whether it is applicable on remote file systems as disccused here http://stackoverflow.com/questions/575328/fcntl-lockf-which-is-better-to-use-for-file-locking and in your post. Do you have any certain info on whether it is fine with networked file system? — Daniel, Jun 29 '13 at 03:16
Only trust what the documentation promises (and I make none; never used it) ;-) — user2246674, Jun 29 '13 at 03:36

score 0 · Answer 1 · answered Jun 29 '13 at 02:28

0

Just a thought...

Couldn't you put a 'lock' file in the same directory as the file your trying to write to? In your distributed processes check for this lock file. If it exists sleep for x amount and try again. Likewise, when the process that currently has the file open finishes the process deletes the lock file?

So if you have in the simple case 2 processes called A and B:

Process A checks for lock file and if it doesn't exist it creates the lock file and does what it needs to with the file. After it's done it deletes this lock file.

If process A detects the lock file then that means process B has the file, so sleep and try again later....rinse repeat.

answered Jun 29 '13 at 02:28

Arash Sharif

542
7
16

1

I guess the problem with this is that the process of creating a file is not atomic, so there's odd that both A and B are creating check files at the same time. – Daniel Jun 29 '13 at 02:33
if you could store this 'file' in a db of some sort, then you could lock at the db level. Not sure if that helps. – Arash Sharif Jun 29 '13 at 02:37
You could also use memcached to store the lock in I guess – Arash Sharif Jun 29 '13 at 02:39
I don't think I have access to a DB. Can you elaborate more on the memcached approach? – Daniel Jun 29 '13 at 02:41
http://memcached.org/ The idea would be put a key in the cache to resemble the lock for example lock=true, lock=false. Since this is in memory it should help if your worried about the time it takes to write the write file to disk. – Arash Sharif Jun 29 '13 at 02:42
if you don't have access to memcached either, then another somewhat silly approach would be to rename the file during use....so that it takes the name of the process thats using it. Then the other process doesn't see it. Someway, somehow you need a semaphore. – Arash Sharif Jun 29 '13 at 02:46
your absolutely right user2246674...specially across network calls I suppose. – Arash Sharif Jun 29 '13 at 02:55
Unlike file, creating a directory is usually atomic. – Evgen Jan 04 '22 at 22:21

Chand Priyankara · Answer 2 · 2013-06-29T06:20:28.023

There are file locking mechanism to multiple process, even written from multiple languages. Operating System specific locking mechanisms are used for this. Java JNI, C++, etc languages have implemented these locking mechanisms to synchronize file access within multiple OS Processes[within LTs-Lightweight Threads].

Look in to your Language specific Native file synchronization mechanisms for this.

Following is a Java based Sample:

FileInputStream in = new FileInputStream(file);
try {
    java.nio.channels.FileLock lock = in.getChannel().lock();
    try {
        Reader reader = new InputStreamReader(in, charset);
        ...
    } finally {
        lock.release();
    }
} finally {
    in.close();
}

This locking should be OS independent [work in Unix like systems, Windows, etc].

For this kind to scenarios, I suggest to use Double Locking for better access controlling.

.. "should be OS independent" .. but this task is not, even if [Java tries to fake it](http://stackoverflow.com/a/2480235/2246674). Furthermore, in general, it is [*not* reliable over a network FS](http://stackoverflow.com/questions/416285/java-file-locking-on-a-network). — user2246674, Jun 29 '13 at 21:42

is there a way to synchronize write to a file among different processes (not threads)

2 Answers2