0

I'm having an issue where I have 8x slower access to a set of files as compared to the same files in another directory on a linux machine.

The filesystem is a 36TB RAID-5 filesystem exported from a Dell PERC H810, and it's formatted with ext4. The machine has 256GB RAM, and I'm using OpenSuSE 12.3 with kernel 3.7.10-1.45-desktop.

The problem is seen with something simple like "time cat slowdir/* > /dev/null", but "time cat fastdir/* > /dev/null" is about 8x faster. I'm clearing the IO cache between tests (echo 3 > /proc/sys/vm/drop_caches), so that shouldn't be affecting my results.

Both slowdir and fastdir are in the same filesystem and same parent directory.

Here are some more oddities about the problem. If I do the following, the problem persists in the new directory, alsoslowdir:

  • cd /parentdir
  • cp -r slowdir alsoslowdir
  • echo 3 > /proc/sys/vm/drop_caches
  • time cat alsoslowdir/* > /dev/null (BAD: takes 8 mins)

But, if I make a new directory, alsofastdir, and copy all of the files into it, then it's 8x faster with this method:

  • cd /parentdir
  • mkdir alsofastdir
  • cp slowdir/* alsofastdir/
  • echo 3 > /proc/sys/vm/drop_caches
  • time cat alsofastdir/* > /dev/null (GOOD: takes 1 min)

All of the files are between 7 and 15MB in each of the directories and there are a few thousand files, with a total of 58GB in the dir.

I've checked the /usr/sbin/filefrag statistics for all of the files in the fast and slow directories and they are all either 1 or 2 extents, with about the same count of 1 & 2 extents between them.

What am I missing?

John Kirkland
  • 1
  • 1
    What is the file count and total size for each directory (your question is not clear which the count/size you've posted is referring to) -- give us both, *accurately*. – fukawi2 Mar 13 '15 at 04:17
  • `ls -la | wc -l` for both - the slow and fast variants? – the-wabbit Mar 20 '15 at 21:54

2 Answers2

0

In your "fast" test, you showed that you copied recursively: cp -r slowdir alsoslowdir

In your "slow" test, you didn't copy with a recursive flag, but rather with a wildcard: cp slowdir/* alsofastdir/

Do you have any sub-directories in slowdir? Not 100% sure whether wildcard also includes sub-directories, but I'm fairly certain it does not, and only expands out to all matching "objects" in the directory, meaning that sub-directories would be left empty.

You may want to retry your test, and ensure with absolute certainty that the data contained by each directory is identical.

If that gets you nowhere... Perhaps just add "fast" to all of your directory names? (j/k) Look into finding a good performance testing tool though - cat really isn't a good method to measure IMO. Find a tool that allows for adjusting threads, I/O size, read/write mix, etc with tests being run on a specific file (sorry, no specific tool names come to mind at the moment).

By the way - what even led you to test performance on individual directories like you're doing? I'm sure there was some kind of odd behavior you encountered to start this off...

JimNim
  • 2,776
  • 13
  • 24
0

Granted that directories structure is absolutely the same (and you should really check that), your results can be skewed by caching (both at controller and OS level) and/or much different physical disk layout for the two directories.

Try that:

  1. make sure that you have iostat binary (it's typically part of sysstat package)
  2. invalidate PERC cache by issuing
    dd if=/dev/zero of=bigfile bs=4k count=1M oflag=direct; sync
  3. invalidate OS cache by issuing
    sync; echo 3 > /proc/sys/vm/drop_caches
  4. gather disk statistics issuing
    iostat -x -k 5 > stat.txt & cat dir/* > /dev/null; killall iostat
  5. repeat all these steps for the other directory, upload you disk statistics (for both directories) and let me see them
shodanshok
  • 47,711
  • 7
  • 111
  • 180
  • He did drop_caches already. And given the directory size, controller caches are not ging to make an 800 percent difference. – the-wabbit Mar 20 '15 at 21:51
  • Without a leading sync, drop_cache drops the read buffers only. Dirty (written and not committed) buffers will remain in their place into memory. Moreover, my reply was about more than drop_cache itself. The point is that without profiling, we can not really help. Anyway, the most probable thing is that the slow directory is misaligned in respect to RAID geometry, but this is a guess only. – shodanshok Mar 21 '15 at 09:49