0

I ask for your help. I myself can no longer see the way out.

I have a server
1x Xeon E-2236
32RAM
4 x 10TB disks (WDC WD101KRYZ-01) - collected in RAID-0.
The disks are fast. Each individually provides 250MB/s read/write. The RAID-0 array produces 900MB / s - which is quite enough to pump a 7 gigabit channel. Test conducted with hdparm -t /dev/md0

The server is designed to distribute large video files - it is a streaming video server.

When I turn on traffic to this server, everything goes well for a while, but over time, iowait grows to its maximum. The HDDs only handle 40 requests per second - which is very small. iowait is busy worker process(nginx).

1. Why? What am I doing wrong?

I tried enabling aio for multithreading in nginx, it helped but not essential.

I increased readahead = 8192, blockdev --setra 65536 - no result.

2. Why does the kernel not try to cache static data in RAM at maximum CPU load?

I checked the disks. There are no bad sectors. HDDs - great.

3. Why does atop show a high load on only one disk? (see screenshot)
4. Shouldn't RAID-0 parallel the load across all disks?

I ask for your help. Thank you!

Grey Hill
  • 1
  • 1
    Spinning rust has low IOPS, which is far more important than sequential read speed. You would have done much better with ten 4TB disks. And spent less money. – Michael Hampton Jun 27 '21 at 14:13
  • I have a second exactly the same server, with the same disks. Works perfect. The only difference is in RAID. RAID-10 on the old server. On a new RAID-0 server. RAID-0 should be faster than 10 - right? – Grey Hill Jun 27 '21 at 14:28
  • Hdparm is never a reliable source for benchmark.And Michaels words are true. Spinning Storage is really low on iops – djdomi Jun 27 '21 at 16:49
  • By the way you can use caching to get a better performance – djdomi Jun 27 '21 at 16:58
  • Djdomi, Please, tell me in what way do you propose to do this? – Grey Hill Jun 27 '21 at 17:11
  • Is the RAID 10 the same quantity of disks? The same quantity of disks in RAID 10 and 0 will give the same read rate (4x). I wonder if adding an SSD and making it swap would effectively increase IOPS? – Paul Jun 28 '21 at 14:52

2 Answers2

0

4 x 10TB disks (WDC WD101KRYZ-01) - collected in RAID-0

First, let me warn against RAID0 unless you are really sure that losing all data in the face of a bad disk is fine. To tell the truth only scratchpad, temporary or very easily reconstructed data should ever be put on a RAID0 array.

The server is designed to distribute large video files - it is a streaming video server

When sequentially read large files, where "large" really means "larger than RAM cache", caching is ineffective: being impossible to cache all file data in RAM, each new re-read of the same file will cause previously cached data to be dropped, effectively continuously trashing your own cache.

The HDDs only handle 40 requests per second - which is very small

Unfortunately, mechanical HDD are very slow for small, random IOs: an healthy 7200 RPM HDD provides ~70 IOPS on average when tested in isolation, probably something less when put in a RAID array.

That said, a streaming service should not hammer the disks with small, random IOs. Can you provide the output of the following commands during such an high-iowait case?

  • lsblk
  • dstat -d -n -f
  • iotop -a
  • iostat -x -k 1
shodanshok
  • 47,711
  • 7
  • 111
  • 180
-2

There's only one solution!

If you need good disk speed, RAID 0 and 5 should not even be considered. Personally tested)) Yes on RAID 0 and 5 you will have a lot of free space, but the speed will be very low. Under high load your server will ask for help.

I installed a software RAID 10 and my problems are gone.

The downside to RAID 10 is that it is expensive. But if you are lucky you will find a hoster with a good price for servers.

Good luck everyone!

P.S. The question is closed!

Grey Hill
  • 1
  • 1
    **This is certainly wrong**. RAID0 is exactly for data loss at very high linear speed, due to its parallelization and lack of redundancy. Optimal state RAID5 and RAID6 should display quite high read speeds, just a little less than RAID0, because they do the same parallelization. Your problem is somewhere else, not in just RAID level, your tests are likely faulty. Please, don't misguide others. – Nikita Kipriyanov Sep 16 '21 at 09:37
  • I concur with @NikitaKipriyanov here - RAID0 is faster. There has to be something wrong with either the software and/or hardware for RAID10 to be faster. Most likely software/config, since you get fair speed with RAID10. – OnkelJ Sep 16 '21 at 10:28
  • For a read only such as video distribution, 10 may very well be faster, as the *same data* can be read from two disks in paralell. – vidarlo Sep 16 '21 at 11:40
  • @vidario how is that possible? I'd rather believe RAID10 will show exactly same performance as RAID0 of same number of spindles. No more, no less, because there is no reason for it to be more, no reason for it to be less. And, just a little better than RAID5 because one less block of data in each stripe than RAID0. And just a little more better thatn RAID6 because it has two less data blocks in each stripe of the same number of spindles. – Nikita Kipriyanov Sep 16 '21 at 11:47