If you have huge number of files in that directory and the contents of the directory constantly change, the directory entry itself gets fragmented over time. Then when the OS is reading the directory contents, there will be lots and lots of unnecessary disk seeks. This happens especially with ext* filesystems (ext4 might be better though) and the old ReiserFS v3.x filesystems (if that got past 85% full or so).
The solution is quite easy:
cp -pr origdir newdir
mv origdir origdir.bak
mv newdir origdir
Of course if everything is cached in RAM, this does not matter that much; usually Linux caches frequently accessed files and dirs quite aggressively. If you truly want to keep the contents of those directories in RAM, you can put something like ls -lah /your/dir 2>&1 >/dev/null
to your cron.
EDIT: Oh, one thing popped on to my mind. If your server has a battery-backed up RAID controller with some cache in it, please check that the battery is OK. I've seen situations where the battery is dead and the controller disables the cache completely, ruining the performance very bad. For example HP servers might tell in the iLO logs something about the controller battery; in the actual server health dashboard everything seems to be fine and green, but only the log entry will tell you about this.