0

Background

I've been having some file locking issues with running Dovecot IMAP storage on a triple-replicated Gluster Volume. The volume is mounted with the glusterfs type using defaults. This implies a FUSE mount. The OS is Fedora 28, kernel 4.18.18-200.fc28.x86_64.

<host>:/volume        /mount                  glusterfs       defaults,x-systemd.automount,noauto     0 0

File lock errors appear in the logs on the dovecot.index* files when moving a large amount of mail in my e-mail client (Thunderbird). Researching the issue I found some Dovecot documentation regarding shared filesystems.

Question

The suggestions made about the lock_method setting seem clear to me. I have specific question regarding the Memory Mapping section:

By default Dovecot mmap()s the index files. This may not work with all clustered filesystems, and it most certainly won't work with NFS. Setting mmap_disable = yes disables mmap() and Dovecot does its own internal caching. If mmap() is supported by your filesystem, it's still not certain that it gives better performance. Try benchmarking to make sure.

Am I affected by this? Does FUSE+GlustFS support mmap() in a proper way?

Research

When poking around on the internet trying to find the answer I found the following havoc:

  • Some issues with a python dev on SO trying to use mmap() in shared mode.
  • A closed PR at Numpy. trying to fix the mmap() interface.

But these are both python related and I'm not sure how Dovecot implements mmap(). I also found a patch over at lwm.net:

From:    Tejun Heo <tj@kernel.org>
To:      Miklos Szeredi <mszeredi@suse.cz>, lkml <linux-kernel@vger.kernel.org>,
    fuse-devel@lists.sourceforge.net
Subject: [PATCH] FUSE/CUSE: implement direct mmap support
Date:    Tue, 09 Feb 2010 15:08:36 +0900
Cc:      Lars Wendler <polynomial-c@gentoo.org>,
    Andrew Morton <akpm@linux-foundation.org>

Implement FUSE direct mmap support. The server can redirect client mmap
requests to any SHMLBA aligned offset in the custom address space attached
to the fuse channel. The address space is managed by the server using
mmap/munmap(2). The SHMLBA alignment requirement is necessary to avoid
cache aliasing issues on archs with virtually indexed caches as FUSE
direct mmaps are basically shared memory between clients and the server

But this addressed FUSE as a whole and doesn't really answer my question with certainty.

Tim
  • 123
  • 1
  • 6

1 Answers1

1

In general GlusterFS supports mmap() just fine. It would be a bug in case something goes wrong with it. GlusterFS has several performance features that can be troublesome for multi-threaded or multi-client workloads. You may want to experiment with disabling all or some of these options:

  • performance.read-ahead
  • performance.write-behind
  • performance.readdir-ahead and performance.parallel-readdir
  • performance.quick-read
  • performance.stat-prefetch
  • performance.io-cache

For example: gluster volume set ${VOLUMENAME} performance.read-ahead off

On the Dovecot wiki page that you linked, there is a paragraph about FUSE/GlusterFS:

FUSE caches dentries and file attributes internally. If you're using multiple GlusterFS clients to access the same mailboxes, you're going to have problems. Worst of these problems can be avoided by using NFS cache flushes, which just happen to work with FUSE as well:

mail_nfs_index = yes mail_nfs_storage = yes

These probably don't work perfectly.

I don't know why it would not work perfectly... It would be good to have some details about that. For most workloads GlusterFS behaves pretty much the same as NFS, so the recommendations would be the same too.

Niels de Vos
  • 146
  • 3