How to simulate INotify failure in functional test?

Question

I have a Linux application that uses inotify for tracking filesystem changes. And I want to write a functional test suite for it that tests the application from the end-user perspective and as part of it I'd like to test situations where filesystem fails and particularly I want to test inotify failure.
Speicifically I'd like to make inotify_init(), inotify_add_watch(), inotify_rm_watch() calls and call of read() for the inotify file-descriptors return an error when it's required in the tests.

But the problem is that I can't find the way of how to simulate inotify failure. I wonder if somebody already encountered such problem and knows some solutions.

Answer 1

If you want to avoid any mocking whatsoever, your best bet is simply provoking errors by directly hitting OS limits. For example, inotify_init can fail with EMFILE errno, if the calling process has reached it's limit on number of open file descriptors. To reach such conditions with 100% precision you can use two tricks:

1. Dynamically manipulate limits of running process by changing values in procfs
2. Assign your app process to dedicated cgroup and "suspend" it by giving it ~0% CPU time via cgroups API (this is how Android throttles background apps and implements it's energy-saving "Doze" mode).

All possible errors conditions of inotify are documented in man pages of inotify, inotify_init and inotify_add_watch (I don't think that inotify_rm_watch can fail except for purely programming errors in your code).

Aside from ordinary errors (such as going over /proc/sys/fs/inotify/max_user_watches) inotify has several fault modes (queue space exhaustion, watch ID reuse), but those aren't "failures" in strict sense of word.

Queue exhaustion happens when someone performs filesystem changes faster than you can react. It is easy to reproduce: use cgroups to pause your program while it has an inotify descriptor open (so the event queue isn't drained) and rapidly generate lots of notifications by modifying the observed files/directories. Once you have /proc/sys/fs/inotify/max_queued_events of unhandled events, and unpause your program, it will receive IN_Q_OVERFLOW (and potentially miss some events, that didn't fit into queue).

Watch ID reuse is tedious to reproduce, because modern kernels switched from file descriptor-like behavior to PID-like behavior for watch-IDs. You should use the same approach as when testing for PID reuse — create and destroy lots of inotify watches until the integer watch ID wraps around.

Inotify also has a couple of tricky corner-cases, that rarely occur during normal operation (for example, all Java bindings I know, including Android and OpenJDK, do not handle all of them correctly): same-inode problem and handling IN_UNMOUNT.

Same-inode problem is well-explained in inotify documentation:

A successful call to inotify_add_watch() returns a unique watch descriptor for this inotify instance, for the filesystem object (inode) that corresponds to pathname. If the filesystem object was not previously being watched by this inotify instance, then the watch descriptor is newly allocated. If the filesystem object was already being watched (perhaps via a different link to the same object), then the descriptor for the existing watch is returned.

In plain words: if you watch two hard-links to the same file, their numeric watch IDs will be the same. This behavior can easily result in losing track of the second inotify watch, if you store watches in something like hashmap, keyed with integer watch IDs.

Second issue is even harder to observe, thus rarely properly supported despite not even being error mode: unmounting a partition, currently observed via inotify. The tricky part is: Linux filesystems do not allow you to unmount themselves when they you have file descriptors opened them, but observing a file via inotify does not prevent the filesystem unmounting. If your app observes files on separate filesystem, and user unmounts that filesystem, you have to be prepared to handle the resulting IN_UNMOUNT event.

All of tests above should be possible to perform on tmpfs filesystem.

Answer 2

After a bit of thinking, I came up with another solution. You can use Linux "seccomp" facility to "mock" results of individual inotify-related system calls. The upsides of this approach are being simple, robust and completely non-intrusive. You can conditionally adjust behavior of syscalls while still using original OS behavior in other cases. Technically this still counts as mocking, but the mocking layer is placed very deeply, between the kernel code and userspace syscall interface.

You don't need to modify the code of program, just write a wrapper, that installs a fitting seccomp filter before exec-ing your app (the code below uses libseccomp):

 // pass control to kernel syscall code by default
 scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_ALLOW);
 if (!ctx) exit(1);

 // modify behavior of specific system call to return `EMFILE` error
 seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EMFILE), __NR_inotify_init, 0));

 execve(...

Seccomp is essentially a limited interpreter, running extended version of BPF bytecode, so it's capabilities are very extensive. libseccomp allows you to install limited conditional filters (for example comparing integer arguments of system call to constant values). If you want to achieve more impressive conditional behavior (such as comparing file path, passed to inotify_add_watch, to predefined value), you can combine direct usage of seccomp() syscall with kernel bpf() facility to write complex filtering programs in eBPF dialect.

Writing syscall filters might be tedious, and the behavior of program under effects of seccomp does not actually depend on kernel implementation (seccomp filters are invoked by kernel before passing the control to kernel syscall handler). So you may want to combine sparse use of seccomp with more organic approach, outlined in the my other answer.

Answer 3

Probably not as non-intrusive as you would like, but the INotify class from inotify_simple is small. You could completely wrap it, delegate all of the methods, and inject the errors.

The code would look something like this:

from inotify_simple.inotify_simple import INotify

class WrapINotify(object):

    init_error_list      = []
    add_watch_error_list = []
    rm_watch_error_list  = []
    read_error_list      = []

    def raise_if_error(self, error_list):

        if not error_list:
            return

        # Simulate INotify raising an exception
        exception = error_list.pop(0)

        raise exception

    def __init__(self):

        self.raise_if_error(WrapINotify.init_error_list)
        self.inotify = INotify()

    def add_watch(self, path, mask):

        self.raise_if_error(WrapINotify.add_watch_error_list)
        self.inotify.add_watch(path, mask)

    def rm_watch(self, wd):

        self.raise_if_error(WrapINotify.rm_watch_error_list)
        return self.inotify.rm_watch(wd)

    def read(self, timeout=None, read_delay=None):

        self.raise_if_error(WrapINotify.read_error_list)
        return self.inotify.read(timeout, read_delay)

    def close(self):

        self.inotify.close()

    def __enter__(self):

        return self.inotify.__enter__()

    def __exit__(self, exc_type, exc_value, traceback):

        self.inotify.__exit__(exc_type, exc_value, traceback)

With this code, you would, somewhere else do:

WrapINotify.add_watch_error_list.append(OSError(28, 'No space left on disk'))

to inject the error. Of course, you could add more code to the wrapper class to implement different error injection schemes.