Decrease in disk performance after partitioning and encryption, is this much of a drop normal?

Question

I have a server that I only have remote access to. Earlier in the week I repartitioned the 2 disk raid as follows:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/sda1_crypt
                      363G  1.8G  343G   1% /
tmpfs                 2.0G     0  2.0G   0% /lib/init/rw
udev                  2.0G  140K  2.0G   1% /dev
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/sda5             461M   26M  412M   6% /boot
/dev/sda7             179G  8.6G  162G   6% /data

The raid consists of 2 x 300gb SAS 15k disks.

Prior to the changes I made, it was being used as a single unencrypted root parition and hdparm -t /dev/sda was giving readings around 240mb/s, which I still get if I do it now:

/dev/sda:
Timing buffered disk reads: 730 MB in  3.00 seconds = 243.06 MB/sec

Since the repartition and encryption, I get the following on the separate partitions:

Unencrypted /dev/sda7:

/dev/sda7:
Timing buffered disk reads: 540 MB in  3.00 seconds = 179.78 MB/sec

Unencrypted /dev/sda5:

/dev/sda5:
Timing buffered disk reads: 476 MB in  2.55 seconds = 186.86 MB/sec

Encrypted /dev/mapper/sda1_crypt:

/dev/mapper/sda1_crypt:
Timing buffered disk reads: 150 MB in  3.03 seconds =  49.54 MB/sec

I expected a drop in performance on the encrypted partition, but not that much, but I didn't expect I would get a drop in performance on the other partitions at all.

The other hardware in the server is:

2 x Quad Core Intel(R) Xeon(R) CPU E5405 @ 2.00GHz and 4gb RAM

$ cat /proc/scsi/scsi            
Attached devices:
Host: scsi0 Channel: 00 Id: 32 Lun: 00
  Vendor: DP       Model: BACKPLANE        Rev: 1.05
  Type:   Enclosure                        ANSI  SCSI revision: 05
Host: scsi0 Channel: 02 Id: 00 Lun: 00
  Vendor: DELL     Model: PERC 6/i         Rev: 1.11
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: HL-DT-ST Model: CD-ROM GCR-8240N Rev: 1.10
  Type:   CD-ROM                           ANSI  SCSI revision: 05

I'm guessing this means the server has a PERC 6/i RAID controller?

The encryption was done with default settings during debian 6 installation. I can't recall the exact specifics and am not sure how I go about finding them?

Thanks

Update:

Ok, it seems I jumped in and repartitioned and encrypted without knowing all the details. This disk really does need encrypting.

I known now that the paritions should be reordered with the encrypted partition at the end of the disk.

Can anyone give me some advice on disk encryption schemes that have a good performance / security balance or point me in the direction of any kind of benchmarking that has been done?

I've had a bit of a google and I'm not turning much up that is helpful. It seems the older Xeons just aren't up to AES disk encryption.

I checked another server with a single Intel(R) Xeon(R) CPU X3430 @ 2.40GHz, and they still maintain 87mb/s, using the exact same encryption scheme on a typical single SATA drive which maxes at 107mb/s without encryption.

I am still investigating fio, seems it's not that straight forward. Most of the examples given are for random access, which isn't what I'm after. I need raw throughput on large files.

Answer 1

Well, the default Debian uses AES encryption by default, to answer that part of your question.

And like symcbean suggests in his comment, get a proper benchmarking tool to measure the performance hit.

However, seeing your read speed drop from 250 MB/s to about 50 MB/s isn't that unusual. So the answer to your question is "yes, that much of a drop is normal for partition encryption... but use some better benchmarking tools to make sure your measurements are accurate."

(And just in general, you should think twice about encrypting a server, and make sure you have a real need before inflicting that on yourself.)

Answer 2

Well, first off is that hard drives are circular. The first sector is on the outside edge of the platter and from there sectors move inward. For the same rotational speed, a section 1.5" from the center of the platter has a linear length of 4.71". A section 1" from the center of the platter has a linear length of 3.14". Since platter spin speed is constant, sectors on the outside track can perform linear reads faster... between those two example tracks, the first one will read 1.5 times faster.

When you partitioned your drive, your unencrypted partitions that rely primarily on drive speed are near the end of the drive's space. This makes them slower than for your first test that was near the beginning of the drive.

Meanwhile, your encrypted partition that relies mainly on CPU speed for throughput (decryption function) is near the outside / start and thus has a faster disk read that doesn't matter because the encryption function is slower than it.

In summary, these speeds makes sense to me. In cases where you have encrypted and unencrypted partitions, it may make sense to order the encrypted partitions to the end of the disk.