0

I'm troubleshooting a linux server's performance degradation from last night. I have sysstat installed on the server and it's polling every 2 minutes. At around the time the event happened, there was a lot of disk activity (using sar -b and sar -d):

05:46:01 PM       tps      rtps      wtps   bread/s   bwrtn/s
05:46:01 PM    246.26      1.85    244.41    141.65  10524.53

05:58:01 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
05:46:01 PM    dev8-0    246.26    141.65  10524.53     43.31      0.09      0.38      0.16      3.92

As you can see, most of it is write access. What I would like to find out is what file(s) were being written to at this time. It doesn't appear that sar maintains this data so I was wondering if there are any other utilities that would monitor this type of activity. I doubt there is any way I can get anything historical, but if I could get something installed now it might help for the next incident.

Ken S.
  • 479
  • 5
  • 14

1 Answers1

1

iotop - it will help you identify process and then you can use lsof -c <process> to list all opened file descriptors opened by process. Or you can strace process to see all current low level syscalls.

These utilities are not for monitoring, they are for real time investigation. You will need to implement proper monitoring, if you need to see also some historic values.

Personal recommendation:

Sysdig (cloud) - https://github.com/draios/sysdig/wiki/Sysdig-Examples#disk:

  • See the top processes in terms of disk bandwidth usage

    sysdig -c topprocs_file

  • List the processes that are using a high number of files

    sysdig -c fdcount_by proc.name "fd.type=file"

  • See the top files in terms of read+write bytes

    sysdig -c topfiles_bytes

  • Print the top files that apache has been reading from or writing to

    sysdig -c topfiles_bytes proc.name=httpd
Jan Garaj
  • 879
  • 1
  • 7
  • 15
  • proper monitoring is precisely why i post the original question. do you have any recommendations on utilities that would monitor which files are being accessed? – Ken S. Jul 21 '16 at 17:52