Correctly close file descriptor when opened through network mount

Question

I'm currently trying to figure out how to correctly close a file descriptor when it points to a remote file and the connection is lost.

I have a simple example program which opens a file descriptor on a sshfs mount folder and start to write to the file.

I'm not able to find how to handle the case when the connection is lost.

void *write_thread(void* arg);

int main()
{
    pthread_t thread;
    int fd = -1;

    if(-1 == (fd = open("/mnt/testfile.txt", O_CREAT | O_RDWR | O_NONBLOCK, S_IRWXU)))
    {
        fprintf(stderr, "Error oppening file : %m\n");
        return EXIT_FAILURE;
    }
    else
    {
        if(0 > pthread_create(&thread, NULL, write_thread, &fd))
        {
            fprintf(stderr, "Error launching thread : %m\n");
            return EXIT_FAILURE;
        }
        fprintf(stdout, "Waiting 10 seconds before closing\n");
        sleep(10);
        if(0 > close(fd))
        {
            fprintf(stderr, "Error closing file descriptor: %m\n");
        }
    }
}

void *write_thread(void* arg)
{
    int fd = *(int*)arg;
    int ret;

    while(1)
    {
        fprintf(stdout, "Write to file\n", fd);
        if(0 > ( ret = write(fd, "Test\n", 5)))
        {
            fprintf(stderr, "Error writing to file : %m\n");
            if(errno == EBADF)
            {
                if(-1 == close(fd))
                {
                    fprintf(stderr, "Close failed : %m\n");
                }
                return NULL;
            }
        }
        else if(0 == ret)
        {
            fprintf(stderr, "Nothing happened\n");
        }
        else
        {
            fprintf(stderr, "%d bytes written\n", ret);
        }
        sleep(1);
    }
}

When the connection is lost (i.e. I unplug the ethernet cable between my boards), The close in the main thread always blocks whether I use the flag O_NONBLOCK or not.

The write call sometimes immediately fails with EBADF error or sometimes continues for a long time before failing.

My problem is that the write call doesn't always fail when the connection is lost so I can't trigger the event into the thread and I also can't trigger it from the main thread because close blocks forever.

So my question is : How to correctly handle this case in C ?

In my experience, it is impossible to handle this situation correctly. Most remote-mount protocols (NFS in particular; I don't know about sshfs) seem to be quite incapable of gracefully handling connection loss. All hell breaks loose. — Steve Summit, Oct 08 '19 at 13:55
You are calling close() on the same fd in 2 threads, and you have no synchronization to prevent race conditions between your threads. I suggest you rather decide which of the two threads should close() the file descriptor, and make sure you only do it in one of the threads. — nos, Oct 08 '19 at 14:00
@SteveSummit I feared this answer. Do you know a filesystem or a network protocol which would support connection loss handling ? — Arkaik, Oct 08 '19 at 14:15
@nos This is for example purpose, I'll decide which thread shall close the fd depending on the best implementation — Arkaik, Oct 08 '19 at 14:17

score 2 · Answer 1 · answered Oct 08 '19 at 14:16

question is: how to correctly handle this case in C?

Simply you can not. File handles are designed to be unified and simple, no matter where they point to. When a device is mounted, and the connection (physical or virtual) to it crashes down, things become tricky even at the command line level.

score 1 · Answer 2 · answered Oct 08 '19 at 17:21

There is a fundamental problem with remote filesystems, where on the one hand you have to cache things in order for performance remain at a usable level, and on the other hand caching in multiple clients can lead to conflicts that are not seen by the server.

NFS, for example, chooses caching by default and if the cache is dirty, it will simply hang until the connection resumes.

Documentation for sshfs suggests similar behavior.

From grepping sshfs' source code, it seems that it doesn't support O_NONBLOCK at all.

None of that has anything to do with C.

IMO your best option is to switch to nfs, and mount with e.g. -o soft -o timeo=15 -o retrans=1.

This could cause data corruption/loss in certain situations when there is a network disconnect, mainly when there are multiple clients or when the client crashes, but it does support O_NONBLOCK and in any case will return EIO if the connection is lost while a request is in-flight.

FWIW the sshfs source code has a function named `sftp_error_to_errno()` which seems to imply that it can return `ECONNABORTED` if the connection is lost while a system call is being served, but I don't have enough experience with ssh to know for sure. — root, Oct 08 '19 at 17:22

score 0 · Accepted Answer · answered Oct 09 '19 at 10:38

After some diggin around I found that the SSH mount could be configured to drop the connection and disconnect from server if nothing happens.

Setting ServerAliveInterval X on client side to disconnect if the server is unresponsive after X sec.

Setting ClientAliveCountMax X on server side to disconnect if the client is unresponsive after X sec.

ServerAliveCountMax Y and ClientAliveCountMax Y can also be used in order to retry Y times before dropping the connection.

With this configuration applied, the sshfs mount is automatically removed by Linux when the connection is unresponsive.

With this configuration, the write call fails with Input/output error first and then with Transport endpoint is not connected.

This is enough to detect that the connection is lost and thus cleaning up the mess before exiting.

Correctly close file descriptor when opened through network mount

3 Answers3