1

I am running FreeBSD 12-RELEASE on our local NFS server. Previously it was working fine, but as of a few days ago some of our tools that require file locking stopped working. Locking does work on our NFS clients for other servers.

Here is some information I think might be relevant:

On our FreeBSD server, san6:

    root@san6:~ # lsof | grep rpc.lockd
    lsof: WARNING: compiled for FreeBSD release 12.0-RELEASE-p5; this is 12.0-RELEASE.
    rpc.lockd 1063  root  cwd   VDIR                 2,16               1024        2 /
    rpc.lockd 1063  root  rtd   VDIR                 2,16               1024        2 /
    rpc.lockd 1063  root  txt   VREG                 2,16              73176 31059299 /usr/sbin/rpc.lockd
    rpc.lockd 1063  root  txt   VREG                 2,16             148320 12279169 /libexec/ld-elf.so.1
    rpc.lockd 1063  root  txt   VREG                 2,16              49560 31070185 /usr/lib/librpcsvc.so.5
    rpc.lockd 1063  root  txt   VREG                 2,16            1949672 46066959 /lib/libc.so.7
    rpc.lockd 1063  root    0u  VCHR                 2,18             0t1091      530 /dev/pts/0
    rpc.lockd 1063  root    1u  VCHR                 2,18             0t1091      530 /dev/pts/0
    rpc.lockd 1063  root    2u  VCHR                 2,18             0t1091      530 /dev/pts/0
    rpc.lockd 1063  root    3u  unix   0xfffff8013dc176d0                0t0          ->0xfffff8013dec6a38

    root@san6:~ # ps aux | grep lockd
    root  1063    0.0  0.0  48016 13692  -  Ds   11:54       0:00.43 rpc.lockd -d 2
    root  3102    0.0  0.0  11256  2736  1  S+   05:53       0:00.00 grep lockd

    root@san6:/usr/ports # rpcinfo -p canada
       program vers proto   port  service
        100000    4   tcp    111  rpcbind
        100000    3   tcp    111  rpcbind
        100000    2   tcp    111  rpcbind
        100000    4   udp    111  rpcbind
        100000    3   udp    111  rpcbind
        100000    2   udp    111  rpcbind
        100024    1   udp  58683  status
        100024    1   tcp  36401  status
        100021    1   udp  33875  nlockmgr
        100021    3   udp  33875  nlockmgr
        100021    4   udp  33875  nlockmgr
        100021    1   tcp  33610  nlockmgr
        100021    3   tcp  33610  nlockmgr
        100021    4   tcp  33610  nlockmgr

    root@san6:~ # dmesg | tail -n 1
    NLM: failed to contact remote rpcbind, stat = 5, port = 28416

    root@san6:~ # mount | grep nfs
    data0 on /data0 (zfs, NFS exported, local, noatime, nfsv4acls)

And on one of our clients, "canada", running Centos 7.6.1810:

[root@canada ~]# dmesg | tail -n 10
[   24.205026] virbr0: port 1(virbr0-nic) entered disabled state
[  588.552140] FS-Cache: Loaded
[  588.594282] FS-Cache: Netfs 'nfs' registered for caching
[  888.710226] lockd: server san6 not responding, still trying
[ 1265.871541] lockd: server san6 OK
[ 1265.871551] lockd: unexpected unlock status: 7
[ 1415.519506] nfs: server san6 not responding, still trying
[ 1439.592184] nfs: server san6 OK
[ 2137.130503] nfs: server san6 not responding, still trying
[ 2622.383586] nfs: server san6 OK

[root@canada ~]# mount | grep /projects
auto.direct on /projects type autofs (rw,relatime,fd=29,pgrp=15535,timeout=300,minproto=5,maxproto=5,direct,pipe_ino=87674)
san6:/data0/projects on /projects type nfs (rw,nosuid,noatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.252.48.106,mountvers=3,mountport=816,mountproto=udp,local_lock=none,addr=10.252.48.106)

Note that lockd is in "Ds" state, I'm not sure which file it is waiting for disk io from. It seems to enter this state when I try to kill/restart it and also after running for awhile. I've tried restarting the server with debug flags turned on (the -d2 in the ps log) but didn't see any more information in the syslog.

I've also tried adding the server to the trusted zone of one of our Centos 7 clients to see if that helped the "NLM: failed to contact remote rpcbind, stat = 5, port = 28416" but it did not seem to, no change to whether that client could lock files or not.

I greatly appreciate any help with this issue!

UPDATE: I've reproduced the issue on another server, still the same symptoms. I'm still not sure what is going on here.

rocky
  • 51
  • 1
  • 7

1 Answers1

1

I figured it out. I thought that adding the server to a single client's trusted zone in the firewall would fix the client. Instead, it turns out that all the client's need the server's IP/ports whitelisted in every client's firewall. If even one client blocks, it seems to spoil locking for everyone.

On CentOS 7, the command is

firewall-cmd --permanent --zone=trusted --add-source=<server's ip>

if you don't mind trusting all ports from the server. Don't forget to reload the firewall's configuration:

firewall-cmd --reload
rocky
  • 51
  • 1
  • 7