0

I've got a bunch of centos 5 boxes with Intel X-25 drives (x25-m in dev, x25-e in prod, I think). We're seeing severely degraded disk performance on one of our dev boxes (which easily does 5+ gb of writes every day, meaning we write the full drive's worth of data several times a month).

The box in question:

  • Intel x25-m
  • Ext3 (which doesn't support TRIM)
  • centos 5
  • vmware ESXi

Wikipedia mentions that newer versions of hdparm (which centos5 doesn't include) can bulk-TRIM free blocks. This utility also sounds potentially useful: http://blog.patshead.com/2009/12/a-quick-and-dirty-wipersh-fix-for-intel-x25-m.html

Disk write performance has dropped to <1 MB/sec while copying a 300 meg directory on this system, as of a month or so ago -- it used to be able to perform the same copy operation at least 5 times faster.

What can I do to recover performance on this system?

Frank Farmer
  • 206
  • 1
  • 2
  • 13
  • Can you further go into what you mean when you say both "CentOS 5" and "VMware ESXi" in the same bullet list? Are you running CentOS virtualized? (This will dramatically change your performance outcomes.) If so, are you using a raw device mapping or a virtual disk? – jgoldschrafe Nov 30 '11 at 00:20
  • Yes, virtualized. My primary concern is that performance on this system has degraded severely in the last month, even though the configuration hasn't changed in a year. It's being outperformed by systems with 5400 RPM drives at this point. – Frank Farmer Nov 30 '11 at 01:31
  • It doesn't matter what filesystem you're running on the box, your VM will never even try to use TRIM as long as it's accessing a virtual disk it knows nothing about. (ESXi doesn't support TRIM, anyway.) I'm not convinced that TRIM by itself is the source of your problem, though. As suggested in my previous answer, try adjusting your filesystem's partition alignment to make sure it lines up on a 4k boundary (and knock LVM out of the equation while you're at it) and see if that helps your performance at all. – jgoldschrafe Nov 30 '11 at 04:12
  • Yeah, there are large number of reasons TRIM won't work for me under normal circumstances (old linux kernel, ext3, vmware), which is why I'm interested in trying to manually TRIM currently unallocated space, if at all possible, with a tool like wiper.sh. – Frank Farmer Nov 30 '11 at 20:58

2 Answers2

1

Running a high writes VM on an SSD and especially on a virtual disk seems to me an amazing challenge (that's a euphemism for something a lot less complimentary).

Main things that could have been going wrong:

  • Using a sparse "growable" virtual disk.

  • Using a journaling filesystem (bad news for the SSD).

  • Extreme filesystem fragmentation.

  • High rate of metadata updates (lack of 'noatime' for example).

  • Filesystem alignment (more than partition alignment) for both the real filesystem and the virtual filesystem.

  • Too small buffer cache and too frequent use of 'sync' by the app.

  • Small writes (less than 256KiB).

BTW most SSDs simulate 4KiB writes, but the physical write (actually erase) sizes are usually between 256KiB-1MiB.

Guessing wildly the app writing so much to the disk is updating a lot of small records on it, and the whole system setup and app design is wrong for that.

Walex
  • 11
  • 1
  • This is a development server, where we copy/deploy fresh installations of our software for testing several times a day. In the end, this boils down to a `cp -r 300megcodedirectory /path/to/depoytarget`. It's this `cp` which is massively underperforming -- it used to take maybe 3 minutes on this box, but has recently degraded to 15 minutes. You mention fragmentation... that's usually not a problem with ext3 is it? – Frank Farmer Nov 30 '11 at 21:01
0

Are your partitions properly aligned? Most SSDs use 4k blocks, which may have adverse effects on your performance if your partitions start at the default 31.5k (63 block) boundary.

jgoldschrafe
  • 4,395
  • 18
  • 18