0

I have Samba installed on a server, that has 8 2TB HDDs in raid5, os is debian 2.6.32, 64bit.
When I copy a 3.5GB file to the server, the first GB is transferred with about 100MB/s, but afterwards the speed drops slowly to 2-3MB/s. I tried to configure SO_RCVBUF from 8192 to 65536, TCP_NODELAY, IPTOS_LOWDELAY and SO_KEEPALIVE, they don't help.
The raid is hardware, the controller 3ware 9750-8i, it has 500MB cache, the memory size doesn't correspond to the problem because the slowdown happens after 1GB. I've tested the network with iperf, it works at 800Mbps, that's not the bottleneck. Both NIC cards on Windows 7 and on the Debian server are fullduplex, i've checked their status.
After I reboot the server and start to upload a file, the memory usage increases to the size of the transferred file. Don't know if that's normal or not. Any ideas what else to try?

The last comment here is recommending some tweaks:

echo 64 > /sys/block/sdb/queue/max_sectors_kb
echo 512 > /sys/block/sdb/queue/nr_requests
echo deadline > /sys/block/sdb/queue/scheduler
echo 20 > /proc/sys/vm/dirty_background_ratio
echo 60 > /proc/sys/vm/dirty_ratio
blockdev --setra 16384 /dev/sdb

how safe is it to apply these? i have some data that i don't want to lose
EDIT: the slowdown is consistent, when i copy a second file after the first is finished, it has the same behaviour.
EDIT2: the HDDs are Hitachi 2TB 64MB, deskstar 7k3000, sata-III
the server has 8GB RAM 1333Mhz

titus
  • 103
  • 6

2 Answers2

1

Has this problem started recently, i.e did it work before or not?
If yes, can you pinpoint to the time it changed and maybe corrleate to some changes?Anything in logs?Do you have any messages in RAID managemenet interface like failing disks/rebuilds?
What happens if you use let's say sftp or nfs instead of samba?

Sergei
  • 1,226
  • 16
  • 25
  • the server is new, it was just installed – titus Dec 27 '11 at 13:58
  • did you try copy with other protocols to isolate the problem? – Sergei Dec 27 '11 at 14:03
  • it seems that samba is not the problem, the problem is with the raid, I've added the tweaks that 3ware is recommending, but the speed is not any better than 10MB/s when copying 30GB of data. What speed have you exeperienced from raid5 configuration when writing? thanks – titus Dec 28 '11 at 19:21
  • Your HDD alone should provide higher sequental throughput on their own:http://www.digitalversus.com/internal-hard-drive/hitachi-deskstar-7k3000-2-tb-p10440/test.html .Do you have latest firmware on your RAID card? – Sergei Dec 29 '11 at 14:57
0

can you try to copy a file from usb attached disk, or e-sata or direct attached drive? in that case u can test if problem is network related or samba configuration or something else. You can monitor raid with iostat. what type of drives You have? 7200k , 5400k ? how much ram is in server ? cache write back is disabled on drives ? and controller has battery backup?

"the first GB is transferred with about 100MB/s" - that correct ? i don't think.

Guntis
  • 683
  • 1
  • 10
  • 22
  • can you look at server (there must be a disk lights) and how they act? lights behave the same way, when first GB is copied or not? – Guntis Dec 23 '11 at 17:14
  • when I attach a USB HDD to the server and copy a file the transfer speed is 35MB/s. And when copy a file from a folder to another on the server, the transfer speed is 100MB/s the first 1GB and then it slowly goes down to the same 35MB/s and oscillates. – titus Dec 23 '11 at 17:15
  • all the disk lights are blinking – titus Dec 23 '11 at 17:18
  • maybe some drive is failing ? what is read performance ? if you copy large file from server to windows box ? – Guntis Dec 23 '11 at 17:20
  • they are blinking at same speed all the time when copying ? – Guntis Dec 23 '11 at 17:20
  • copying from the server is +60MB/s, that works ok, writing to the server is the problem – titus Dec 23 '11 at 17:22
  • not really each disk blinks when it wants, not in unison, and they don't seem to blink faster when it is accesing more data – titus Dec 23 '11 at 17:24
  • http://www.vmdamentals.com/?p=897 raid 5 is not good for that big array, because rebuild time is slow, and raid 5 write performance is not so good at all... i suggest google bit around and think about it. here is good info http://www.techrepublic.com/blog/datacenter/calculate-iops-in-a-storage-array/2182 – Guntis Dec 23 '11 at 17:39
  • only now i've noticed that while it copies the first GB the HDD LEDs don't blink, they blink only afterwards, it seems that the first GB is cached 512MB in the controller and 8*64MB in the HDD cache – titus Dec 23 '11 at 17:55
  • raid 10 would be too expensive, raid 1 and raid 0 don't apply for me, raid 6 is even worse than raid 5 because it needs to calculate two parities instead of one. – titus Dec 23 '11 at 17:57