Nfs Hanged suddenly ! Not able to even restart service :
# /etc/init.d/nfs-kernel-server restart
* Stopping NFS kernel daemon [ OK ]
* Unexporting directories for NFS kernel daemon... [ OK ]
* Exporting directories for NFS kernel daemon... [ OK ]
* Starting NFS kernel daemon ^C
Syslogs says nfs is blocked for more than 120sec. How to understand the logs ? what has gone wrong here ?
# cat /var/log/syslog | grep nfs
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248354] INFO: task nfsd:2318 blocked for more than 120 seconds.
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248363] nfsd D 0000000000000000 0 2318 2 0x00000000
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248406] [<ffffffffa02a28c0>] ? nfsd_proc_getattr+0xa0/0xa0 [nfsd]
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248418] [<ffffffffa02a3c11>] nfsd_vfs_write.isra.13+0x81/0x3b0 [nfsd]
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248422] [<ffffffffa02bf691>] ? lookup_clientid+0x51/0x80 [nfsd]
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248426] [<ffffffffa02bf766>] ? nfsd4_lookup_stateid+0xa6/0xc0 [nfsd]
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248429] [<ffffffffa02a7773>] nfsd_write+0x93/0x110 [nfsd]
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248433] [<ffffffffa02b2281>] nfsd4_write+0x1b1/0x250 [nfsd]
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248437] [<ffffffffa02b35ca>] nfsd4_proc_compound+0x56a/0x7c0 [nfsd]
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248440] [<ffffffffa029fd2b>] nfsd_dispatch+0xbb/0x200 [nfsd]
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248455] [<ffffffffa029f71f>] nfsd+0xbf/0x130 [nfsd]
Sep 4 10:02:11 InksedgeDev kernel: [2427120.248458] [<ffffffffa029f660>] ? nfsd_destroy+0x80/0x80 [nfsd]
Output of Iostat :
# iostat 5 5
Linux 3.13.0-29-generic (InksedgeDev) 09/04/2015 _x86_64_ (2 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
5.68 0.00 0.48 0.25 0.01 93.58
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
xvdap1 3.63 15.71 82.80 38138709 201066916
xvdb 0.01 0.05 0.01 119271 15231
xvdc 0.01 0.05 0.01 112329 14600
xvdf 4.25 155.14 175.05 376748353 425087092
md0 0.02 0.10 0.01 231288 29831
avg-cpu: %user %nice %system %iowait %steal %idle
0.30 0.00 0.20 49.75 0.00 49.75
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
xvdap1 0.00 0.00 0.00 0 0
xvdb 0.00 0.00 0.00 0 0
xvdc 0.00 0.00 0.00 0 0
xvdf 0.00 0.00 0.00 0 0
md0 0.00 0.00 0.00 0 0
What might be the reason ? how to troubleshoot ?
