2

I am running a web server (Ubuntu 11.04) that exhibits unexpected high write traffic. When the server is not supposed to write at all, the amount of write traffic is comparable to the read traffic.

Concerned about unnecessary write operation, I tried to analyzed what is going wrong on the system. I can exclude heavy apache logging or access time issues (using noatime mount configuration).

To track down the problem I wanted to see what files where written. Therefore I enabled IO loggin via block_dump (useful blog entry on this topic: sprocket.io). Every filesystem activity will get logged in syslog. Here a short excerpt of my system:

Aug 21 18:22:55 xxxxx kernel: [3984721.590864] apache2(2761): READ block 1098502400 on md2 (8 sectors)
Aug 21 18:22:55 xxxxx kernel: [3984721.594005] kjournald(316): WRITE block 2224394648 on md2 (8 sectors)
Aug 21 18:22:55 xxxxx kernel: [3984721.594029] md2_raid1(260): WRITE block 2925532672 on sdb3 (8 sectors)
Aug 21 18:22:55 xxxxx kernel: [3984721.594044] md2_raid1(260): WRITE block 2925532672 on sda3 (8 sectors)
Aug 21 18:22:55 xxxxx kernel: [3984721.644244] apache2(2761): READ block 2242118744 on md2 (8 sectors)

Ok, now I know what blocks were written. But is there a way to actually identify the filenames that were written based on these block ids?

Thanks for your help!

BTW: I am using a Software Raid, might be part of the problem.

linqu
  • 123
  • 3

3 Answers3

2

Assuming ext2/ext3/ext4, start with

[406420.877412] vi(12496): READ block 4522288 on dm-1 (8 sectors)

Determine the filesystem block size:

# /sbin/dumpe2fs /dev/dm-1 | grep 'Block size'
dumpe2fs 1.42.3 (14-May-2012)
Block size:               4096

Assuming you have a drive with 512 byte sectors divide the block by 4096/512, i.e., 8 to get 565286.

In debugfs use a combination of icheck and ncheck:

debugfs:  icheck 565286
Block   Inode number
565286  142506

debugfs:  ncheck 142506
Inode   Pathname
142506  /etc/security/time.conf

Edit: do this on the md* devices, not the sd* devices. The sd* device I/O is the result of software raid.

Mark Wagner
  • 18,019
  • 2
  • 32
  • 47
1

Filesystem resides on a higher level of abstraction than block devices and software RAIDs. That said, software RAID is not a part of the problem with 99.9% probability, it's just a block device. So, you should use other set of tools to analyze your I/O activity. I would recommend to start with iotop to identify the top writer among running processes first. Then you will be able to use lsof and strace to identify what files are being written to.

Alex
  • 7,939
  • 6
  • 38
  • 52
0

Linux has a kernel system called inotify for watching any changes to a file. In userland, one can use the inotify-tools (apt-get install inotify-tools) to watch a directory. Each file has to have an individual watch placed on it, however. You can recursively apply them to a directory (even the root), but the less you watch, the less overhead there is.

Other options for narrowing things down include:

  • atop which will allow you to see which processes are performing writes
  • auditctl which has a very arcane syntax, but allows watches to be placed on any system call
Jeff Ferland
  • 20,547
  • 2
  • 62
  • 85