Since a recent linux kernel upgrade (3.2.78 on Debian Wheezy) on both my NFS clients and NFS server, I'm seeing really high increase (10 to 20 folds) in the number of nfs_lookup requests from the clients to the server. This is making a huge performance degradation to the application running on my NFS clients (apache+php-fpm).
What can be the cause of such a high increase of nfs_lookup?
Output of mountstats --nfs on one of the clients after about 1h of uptime is as follows:
Stats for ruchbah.nfs://membres mounted on /srv/membres:
NFS mount options: rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.74.13,minorversion=0,local_lock=none
NFS server capabilities: caps=0x7fff,wtmult=512,dtsize=32768,bsize=0,namlen=255
NFSv4 capability flags: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3
NFS security flavor: 1 pseudoflavor: 0
Cache events:
data cache invalidated 1231 times
attribute cache invalidated 23693734 times
inodes synced 1790 times
VFS calls:
VFS requested 46296214 inode revalidations
VFS requested 35900771 dentry revalidations
VFS called nfs_readdir() 3678 times
VFS called nfs_lookup() 35200320 times
VFS called nfs_permission() 11201 times
VFS called nfs_file_open() 22547 times
VFS called nfs_file_flush() 1400 times
VFS called nfs_lock() 7611 times
VFS called nfs_fsync() 65321 times
VFS called nfs_file_release() 3731 times
VM calls:
VFS called nfs_readpage() 856 times
VFS called nfs_readpages() 6945 times
VFS called nfs_writepage() 10789 times
VFS called nfs_writepages() 945872 times
Generic NFS counters:
File size changing operations:
truncating SETATTRs: 65272 extending WRITEs: 0
291 silly renames
short reads: 11201 short writes: 17
NFSERR_DELAYs from server: 0
NFS byte counts:
applications read 162084951 bytes via read(2)
applications wrote 19636930 bytes via write(2)
applications read 0 bytes via O_DIRECT read(2)
applications wrote 0 bytes via O_DIRECT write(2)
client read 152501261 bytes via NFS READ
client wrote 20783074 bytes via NFS WRITE
The number of VFS calls (nfs_lookup in particular) is very high. The cache attribute invalidation is also abnormally high. The clients mostly uses the NFS mount to read data.
