mdadm raid5 too slow when writing

Question

I have set up a RAID5 array using 4 disk partition. All disks are 2TB. When the RAID5 array was initially syncing it was too slow. So I did a

echo 32768 > /sys/block/md2/md/stripe_cache_size

and then it was quiet fast. But still it took a days time to finish syncing. Now I want am doing a

dd if=/dev/zero of=/dev/md2 bs=1024k

onto the RAID5 array and it is like a day over and yet it has not finished. Will this take too much time am I doing some thing wrong.

My raid details are here http://dpaste.com/749742/
Iostat details here http://dpaste.com/749761/ hdd partitioning details here http://dpaste.com/749765/

I have an identical machine with the similar details. But that's quiet faster.

UPDATE: I have added my hdd partitioning details.

Answer 1

Normal time to build such array is 5.5-6 hours. There are a few things that can slow down array rebuild process - concurrent IO on the same disk, high cpu usage on the system, faulty HDD or cables.

dd if=/dev/zero of=/dev/md2 bs=1024k - will slow down rebuild. If you really need to rewrite your array with zeroes - do it after array has built.

Check if you have load on other partitions - using atop or iostat -nx 1

Check your load average in atop or top

Check for HDD errors in system logs and with smartctl.

Answer 2

in raid 5 you should look about raid chunk size. run : cat /proc/mdstat to check yours.

to change it run :

mdadm --grow /dev/md0 --chunk=16

this will rebuild your array, it takes long times to complete. you can read this for more informations about it : https://raid.wiki.kernel.org/index.php/RAID_setup#Chunk_sizes

then you can test again your write performance with dd for exemple.

I've made a small script to test read ans write :

#!/bin/bash

MDDEV="`cat /proc/mdstat | grep md | head -1 | awk '{print $1}'`"
if [ -z "$MDDEV" ]
then
    echo "I can\'t find any md" 
    exit 1
fi

DEVS="`cat /proc/mdstat | grep $MDDEV | tr " " "\n" | grep '^sd' | grep -v \(S\)$ | awk '{print substr($0,3,1)}' | tr -d "\n"`"

NUMDEVS=${#DEVS}

# write dd performance
echo 3 > /proc/sys/vm/drop_caches
dd if=/dev/zero of=/dev/$MDDEV bs=1M count=1024 conv=fdatasync,notrunc

for RASIZE in 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 
do
    echo $RASIZE
    MDRASIZE=$[$RASIZE*$NUMDEVS]
    blockdev --setra $MDRASIZE /dev/$MDDEV
    blockdev --setra $RASIZE /dev/sd[$DEVS]
    blockdev --setra $RASIZE /dev/sd[$DEVS]1

    echo 3 > /proc/sys/vm/drop_caches
    hdparm -t /dev/$MDDEV | grep "Timing"
    #echo 3 > /proc/sys/vm/drop_caches
    #dd if=/dev/$MDDEV of=/dev/null bs=1M count=1024 | grep "bytes"
done

for read performance this script will give you the best read ahead value. then put the blocdev command in your /etc/rc.local file to make it persistent

Answer 3

Software RAID-5 is notoriously slow, and as DukeLion has pointed out, any other load on the system - CPU or IO - will slow it down even more. If you must do RAID-5, I would very strongly advise you to do it in hardware (and proper hardware, not one of these cards where the driver does it in software with the system CPU); or shrug, prepare to lose a third of your array, and do software RAID-1 (which is just fine, performance-wise).

Edit: I didn't say RAID-5 was slow; I said RAID-5 in software was slow. If you compare IO stats from your other (decent) system with those from a hardware RAID-5 I think you'll find that the difference is noticeable. I have a new system here with a decent hardware RAID card (Dell H700) and I'd be happy to do some timing tests with you, if you'd like.

I accept that it's frustrating that your two software RAID-5s perform differently, but without knowing a lot more about the physical and logical setup of each, it's difficult to make tuning recommendations, and I'm not sure I'd bother, since tuning software RAID-5 is, in my experience (and please pardon the expression), a bit like gilding a turd.