Using java file locks within single JVM and across multiple JVMs

Question

I guess I miss something, but I cannot understand how file locks work in Java. To be more exact - how it is implemented.

It seems I cannot acquire (even cannot attempt acquiring) two or more locks for the same file inside single JVM. First lock will be successfully acquired, all further attempts to acquire more locks will result in OverlapingFileLockException. Nevertheless it works for separate processes.

I want to implement data-storage backed by file-system which is intended to work with multiple concurrent requests (both read and write). I want to use file locks to lock on particular files in the storage.

It seems that I have to introduce one more synchronization (exclusive) on JVM-level and only then sync on files to avoid this exception.

Did anyone do anything like that?

I prepared simple test case to show what my problem is. I use Mac OS X, Java 6.

import junit.framework.*;

import javax.swing.*;
import java.io.*;
import java.nio.channels.*;

/**
 * Java file locks test.
 */
public class FileLocksTest extends TestCase {
    /** File path (on Windows file will be created under the root directory of the current drive). */
    private static final String LOCK_FILE_PATH = "/test-java-file-lock-tmp.bin";

    /**
     * @throws Exception If failed.
     */
    public void testWriteLocks() throws Exception {
        final File file = new File(LOCK_FILE_PATH);

        file.createNewFile();

        RandomAccessFile raf = new RandomAccessFile(file, "rw");

        System.out.println("Getting lock...");

        FileLock lock = raf.getChannel().lock();

        System.out.println("Obtained lock: " + lock);

        Thread thread = new Thread(new Runnable() {
            @Override public void run() {
                try {
                    RandomAccessFile raf = new RandomAccessFile(file, "rw");

                    System.out.println("Getting lock (parallel thread)...");

                    FileLock lock = raf.getChannel().lock();

                    System.out.println("Obtained lock (parallel tread): " + lock);

                    lock.release();
                }
                catch (Throwable e) {
                    e.printStackTrace();
                }
            }
        });

        thread.start();

        JOptionPane.showMessageDialog(null, "Press OK to release lock.");

        lock.release();

        thread.join();
    }

    /**
     * @throws Exception If failed.
     */
    public void testReadLocks() throws Exception {
        final File file = new File(LOCK_FILE_PATH);

        file.createNewFile();

        RandomAccessFile raf = new RandomAccessFile(file, "r");

        System.out.println("Getting lock...");

        FileLock lock = raf.getChannel().lock(0, Long.MAX_VALUE, true);

        System.out.println("Obtained lock: " + lock);

        Thread thread = new Thread(new Runnable() {
            @Override public void run() {
                try {
                    RandomAccessFile raf = new RandomAccessFile(file, "r");

                    System.out.println("Getting lock (parallel thread)...");

                    FileLock lock = raf.getChannel().lock(0, Long.MAX_VALUE, true);

                    System.out.println("Obtained lock (parallel thread): " + lock);

                    lock.release();

                }
                catch (Throwable e) {
                    e.printStackTrace();
                }
            }
        });

        thread.start();

        JOptionPane.showMessageDialog(null, "Press OK to release lock.");

        lock.release();

        thread.join();
    }
}

score 8 · Answer 1 · answered Apr 20 '11 at 12:21

8

From the Javadoc:

File locks are held on behalf of the entire Java virtual machine. They are not suitable for controlling access to a file by multiple threads within the same virtual machine.

answered Apr 20 '11 at 12:21

user207421

305,947
44
307
483

score 2 · Answer 2 · answered Apr 19 '11 at 11:08

2

You can only acquire a lock once per file. Locks are not re-entrant AFAIK.

IMHO: Using files to communicate between process is a very bad idea. Perhaps you will be able to get this to work reliably, let me know if you can ;)

I would have one and only one thread read/write in only one process.

answered Apr 19 '11 at 11:08

Peter Lawrey

525,659
79
751
1,130

1

That's not entirely true. It is possible to acquire a lock for a specific region of a file. As such, multiple locks can be acquired for a single file, so long as each lock specifies a different, non-overlapping region. – aroth Apr 19 '11 at 11:19
Shared storage seems to be one of the possible ways of communication between processes (that may run on different hosts). No matter what storage you use - S3, database or shared file system. – Yakov Apr 19 '11 at 11:19
1

Using a database, a single process manages the file and you access its files indirectly. I have found trying to update raw files across multiple hosts in a performant and relable way is very difficult. – Peter Lawrey Apr 19 '11 at 11:21
@Yakov, if you don't want to use an SQL database, you can try one of http://nosql-database.org/ – Peter Lawrey Apr 19 '11 at 11:24
"Using files to communicate between process is a very bad idea." @Peter Lawrey I would like to know why? The simplest of the use cases its much lighter than database, JMS anything. Then why is this bad idea? Any literature yu can point to? – manocha_ak Dec 02 '13 at 13:12
@manocha_ak I should qualify that given I have library which uses files for IPC ;). Using the filesystem as a message queue rarely works partularly well. JMS uses files but not to pass messages. Eg the consumer doesn't get a message in a file and delete it when finished. – Peter Lawrey Dec 02 '13 at 18:27
I agree, I should also qualify that what I am thinking of doing doesn't qualify as communication in full or it might, I am not 100% sure. I would like to think of it as inter process shared data. But then processes are reading and writing that data, should I call that communication? Anyways that's my use case.files suitable for this case? FileLock give me impression that I can use file as global lock sort of / can say inter process / inter jvm look. Like I said my use case is that much only. So, why database or JMS? Are there some characteristics which one must look for IPC, File absolute no? – manocha_ak Dec 03 '13 at 12:17
@manocha_ak Instead of locking, a simple approach is to rename the file, possibly a couple of times for each stage. This works best for large data sizes, esp more than 10 MB AFAIK. However, the smaller the messages, the greater the overhead of using raw files. – Peter Lawrey Dec 03 '13 at 16:29
Great Peter! Thanks to you, I converted my first lock algo to lock free algo. Actually, it feels good doing that, I thought it would be tough all before it. Now the complete answer like you said I made different files, differentiated with pid and then used the deleteOnExit() too, did away with FileLock. My file size will remain in Kbs only, I am pretty sure about it. Max 1-2MB. – manocha_ak Dec 04 '13 at 09:07
Also unearthed a problem with File Steam constructor which uses the FileDescriptor as argument, it doesn't overwrite a file, no way to do that do. Not sure if I am missing something? – manocha_ak Dec 04 '13 at 09:34
@manocha_ak Missing what, exactly? That constructor assumes you already have the file you need in the state you need it in, and just wraps the FD in a stream. If you want a new file, don't use that constructor. – user207421 Dec 11 '13 at 21:40
Yeah I solved it later. Being relatively new to nio was the issue. I got it myself later. – manocha_ak Dec 12 '13 at 05:37

score 2 · Answer 3 · answered Apr 19 '11 at 11:17

2

Have you checked the documentation? The FileChannel.lock() method returns an exclusive lock across the entire file. If you want to have multiple locks active concurrently across different threads, then you cannot use this method.

Instead you need to use FileChannel.locklock(long position, long size, boolean shared) in order to lock a specific region of the file. This will allow you to have multiple locks active at the same time, provided that each one is applied to a different region of the file. If you attempt to lock the same region of the file twice, you will encounter the same exception.

answered Apr 19 '11 at 11:17

aroth

54,026
20
135
176

I want locks across entire file (read or write, depending on situation), since I have multiple files in the storage (each file is rather small). BTW, aroth, did you see my snippet. it seems I will never understand the logic of the guy implemented locks in JDK. WHY (!) it is possible to acquire read lock in the multiple processes or exclusive lock will block the thread if I try to get it in another process, but when I take the lock in the same JVM i get exception. when I call lock() I expect current thread to block until lock is acquired. I can use tryLock if I dont want to block. – Yakov Apr 20 '11 at 08:00
@Yakov - I get what you mean, it's confusing to have the exception be thrown instead of just letting the caller block waiting for the lock. But maybe you can work around the issue with something like `while ((lock = chan.tryLock()) == null) { Thread.sleep(100); }`? Or maybe you could use a pattern like `synchronized(chan) { lock = chan.lock(); //... }` in order to prevent calling `lock()` twice in the same JVM? – aroth Apr 20 '11 at 08:33
tryLock is not always working - sometimes I get NPE somewhere inside FileLockImpl (sources are not included in JDK) when trying to release lock. So, I think, that the only solution here is to add JVM-level synchronization. Thanks. – Yakov Apr 20 '11 at 08:53
There is also one more problem. JVM synchronization maybe very tricky when objects' classes that need file locks are loaded with different class loaders (e.g. different apps in tomcat or similar). – Yakov Sep 14 '11 at 16:42
True but doesn't address the real problem, which is that locks are held by processes, not threads. – user207421 Jan 26 '17 at 22:22

Using java file locks within single JVM and across multiple JVMs

3 Answers3

Linked