How to diagnose intermittently slow raid data transfer rate?

Question

I have an encrypted RAID6 set up on Centos 5.5. When I transfer a large 10gb file from the raid to my computer it will pause at around 4-5gb and again at 9gb. The data transfer drops from 25mb/sec to < 1mb/sec for about 30-60 seconds.

Streaming HD video will stutter and/or freeze every few minutes.

Transfer is via samba. The same thing happens over rsync (using rsync daemon) but the transfer rates are slightly higher.

Can anyone tell me how to go about tracking down the problem? I'm not sure how to find out where the bottleneck is.

Background Info:

Server: CPU is one Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz
Ram: 4gb
Ethernet: gigabit on motherboard
SATA: onboard x8, 6 ports on one controller, 2 on another

CPU usage is about 45% on one core while transferring data from the raid.

How I created the raid:

Raid stack is as follows;

disk-->RAID-->dm-crypt-->LVM-->fs

I have 7x2TB disks. Each have one 'fd' (raid) partition ( /dev/sdb -> /dev/sdh ) note /dev/sda is a SSD containing the OS

Here's how i created it

Create the raid device

mdadm --create /dev/md0 --level=6 --raid-devices=7 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1

open encrypted array

cryptsetup --key-file /mnt/usbstick/img_0768.jpg luksOpen /dev/md0 md0encrypted

Create LVM

pvcreate /dev/mapper/md0encrypted

Create the volume group - use 256M, that allows about 16.7TB. Must be a power of 2

vgcreate -s 256M VolGroupRaid /dev/mapper/md0encrypted

Create the volume. (got the total PE 37260 from vgdisplay -v)

lvcreate -l 37260 VolGroupRaid --name raidvol

Format it using

mkfs.ext3 /dev/VolGroupRaid/raidvol

Mount it

mount /dev/VolGroupRaid/raidvol /mnt/raid

System info

lspci

00:00.0 Host bridge: Intel Corporation 82P965/G965 Memory Controller Hub (rev 02)
00:01.0 PCI bridge: Intel Corporation 82P965/G965 PCI Express Root Port (rev 02)
00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #5 (rev 02)
00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #2 (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 02)
00:1c.4 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 5 (rev 02)
00:1c.5 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 6 (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev f2)
00:1f.0 ISA bridge: Intel Corporation 82801HB/HR (ICH8/R) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801HR/HO/HH (ICH8R/DO/DH) 6 port SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 02)
01:00.0 VGA compatible controller: nVidia Corporation G84 [GeForce 8600 GT] (rev a1)
03:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8053 PCI-E Gigabit Ethernet Controller (rev 22)
04:00.0 SATA controller: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 02)
04:00.1 IDE interface: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 02)

dmesg | grep -i raid

device-mapper: dm-raid45: initialized v0.2594l
md: Autodetecting RAID arrays.
raid5: automatically using best checksumming function: generic_sse
raid5: using function: generic_sse (6816.000 MB/sec)
raid6: int64x1   1558 MB/s
raid6: int64x2   1937 MB/s
raid6: int64x4   1765 MB/s
raid6: int64x8   1468 MB/s
raid6: sse2x1    3308 MB/s
raid6: sse2x2    4863 MB/s
raid6: sse2x4    5582 MB/s
raid6: using algorithm sse2x4 (5582 MB/s)
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
raid5: device sdh1 operational as raid disk 6
raid5: device sdg1 operational as raid disk 5
raid5: device sdf1 operational as raid disk 4
raid5: device sde1 operational as raid disk 3
raid5: device sdd1 operational as raid disk 2
raid5: device sdc1 operational as raid disk 1
raid5: device sdb1 operational as raid disk 0
raid5: allocated 7412kB for md0
raid5: raid level 6 set md0 active with 7 out of 7 devices, algorithm 2
RAID5 conf printout:

cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4] 
md0 : active raid6 sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
      9767559680 blocks level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU]

unused devices: <none>

score 2 · Answer 1 · answered Dec 03 '10 at 00:20

The way to diagnose it is to remove/substitute components. Your full setup is:

 centos(disk-->RAID-->dm-crypt-->LVM-->fs)-->network->mycomputer

Try a different destination:

 centos(disk-->RAID-->dm-crypt-->LVM-->fs)-->network->anothercomputer

Eliminate the network:

 centos(disk-->RAID-->dm-crypt-->LVM-->fs)-->centos

Eliminate all the stacks to see if it is network:

 centos(disk(one disk)-->fs)-->network->mycomputer

If it isn't just the network start adding components:

 centos(disk-->RAID-->fs)-->network->mycomputer
 centos(disk-->RAID-->LVM-->fs)-->network->mycomputer

Those are about all the combinations I can think of.

Thanks! I'm in the process of trying this now. Is there anything better than iostat or an application that shows iostat as graphs I can use? — zio, Dec 04 '10 at 00:20

score 1 · Answer 2 · answered Dec 02 '10 at 21:07

Even your best case transfer rates seem pretty poor, reading a large file from a 7 drive RAID 6 should be able to easily saturate a GigE link, so even with SAMBA overheads you should be seeing much better than 25Meg/sec.

Crypto overhead shouldn't be holding you back as a general principle, encrypting\decrypting (with AES) 100Meg/sec and more is well within the capability of a single core on a modern CPU, however this message about dm-crypt indicates it could be a bottleneck that might be triggered by your setup and it is a relatively recent so I suspect that it is still an issue.

When you say you are transferring to your computer - what is the spec there - What OS, how much RAM and how fast is the drive subsystem you are writing to, ie can it keep up with 25Meg/sec?

The computer I am transferring to is a Macbook Pro running the latest version of OS X. 4GB Ram, the storage I am copying to is a OCZ Vertex 2e SSD. Duplicating a file on the SSD yields 114mb/sec transfer rate. I have also tried to a HDD in the same machine and also to a mac mini on the same lan. — zio, Dec 03 '10 at 00:09
I've corrected my original post. I am using one dual core cpu rather than 2 as originally stated. — zio, Dec 03 '10 at 00:18

How to diagnose intermittently slow raid data transfer rate?

2 Answers2