Docker and file locking

Question

I wrote a simple go application and added a flock system to prevent being running twice at the same time:

import "github.com/nightlyone/lockfile"

lock, err := lockfile.New(filepath.Join(os.TempDir(), "pagerduty-read-api.lock"))
if err != nil {
    panic(err)
}

if err = lock.TryLock(); err != nil {
    fmt.Println("Already running.")
    return
}

defer lock.Unlock()

It works well on my host. On docker, I tried to run it with volume sharing of tmp:

docker run --rm -it -v /tmp:/tmp my-go-binary

But it does not work. I suppose it's because the flock system is not ported on volume sharing.

My question: Does Docker have option to make flock working between running instance? If not, what are my other options to have the same behavior?

Thanks.

Answer 1

This morning I wrote a little Python test program that just writes one million consecutive integers to a file, with flock() locking, obtaining and releasing the lock once for each number appended. I started up 5 containers, each running that test program, and each writing to the same file in a docker volume.

With the locking enabled, the numbers were all written without interfering with each other, and there were exactly 5 million integers in the file. They weren't consecutive when written this way, but that's expected and consistent with flock() working.

Without locking, many of the numbers were written in a manner that indicates the numbers were running afoul of the multitasking sans locking. There were only 3,167,546 numbers in the file and there were 13,357 blank lines. That adds up to the 3,180,903 lines in the file - substantially different than the desired 5,000,000.

While a program cannot definitively prove that there will never be problems just by testing many times, to me that's a pretty convincing argument that Linux flock() works across containers.

Also, it just kinda makes sense that flock would work across containers; containers are pretty much just a shared kernel, distinct pid, distinct file (other than volumes) and distinct IP port space.

I ran my test on a Linux Mint 19.1 system with Linux kernel 4.15.0-20-generic, Docker 19.03.0 - build aeac949 and CPython 3.6.8.

Go is a cool language, but why flock() didn't appear to be working across volumes in your Go program I do not know.

HTH.

Answer 2

I suppose that you want to use docker volume or you may need some other docker volume plugins.

According to this article, Docker Volume File permission and locking, docker volumes only provides a way to define a volume to use by multi containers or use by a container after restarting.

In docker volume plugins, flocker may meet your requirements. Flocker is an open-source Container Data Volume Manager for your Dockerized applications.

BTW, if you are using kubernetes, you may need to learn more about persistent volume, persistent volume claim, storage class.

Answer 3

I've been doing some research on this myself recently, and the issue is that nightlyone/lockfile doesn't actually use the flock syscall. Instead, the lockfile it writes is a PIDfile, a file that just contains the PID (Process IDentifier) of the file that created it.

When checking if a lock is locked, lockfile checks the PID stored in the lockfile, and if it's different to the PID of the current process, it sees it as locked.

The issue is that lockfile doesn't have any special logic to know if it's in a docker container or not, and PIDs get a little muddled when working with docker; the PID of a process when viewed from inside the container will be different from the PID of a process outside the container.

Where this often ends up is that we have two containers running your code above, and they both have PIDs of 1 within their containers. They'll each try to create a lockfile, writing what they think their PID is (1). They then both think they hold the lock - after all, their PID is the one that wrote it! So the lockfile is ignored.

My advice is to switch to a locking implementation that uses flock. I've switched to flock, and it seems to work okay.