9

With the development of hyperconverged systems, we can see other technology than RAID to handle distributed storage (like ceph for example or VSAN or distributed storage like in nutanix).

But is there open source Technology derivate from object storage that I can use on a single server with a lot of big disk, let's say a dozen of 12To sata disk, that will ensure data safety but without the drawback of huge rebuild time when replacing a disk. So Dynamic Disk Pool is a form of de-clustered RAID that can be used at block level like iscsi or filesystem level like zfs or brtrfs.

I would like to provide something similar in feature to a SYNOLOGY bay (NFS, iscsi target, SMB) but without RAID for the block storage.

thanks for your indications.

dominix
  • 446
  • 2
  • 4
  • 13
  • Have you considered (for example) A RAID 5 array with a sixth drive as a hot spare? – Mike Waters May 19 '19 at 19:39
  • 1
    Everyone recommends not to use RAID5 for big volumes. Btw raid5 do not mean 5 disks. But yes hot spare is recommended, however I already have experienced problems with hot spare, sometimes hot spare died because they have been spinning for years before been used. – dominix May 19 '19 at 20:40
  • 1
    You need more disks. The only solution to rebuild delay is more redundancy. Also erasure codes. – Dan D. May 19 '19 at 23:21

2 Answers2

7

For a single system, I would really suggest to stick with well know and performing RAID arrays. Object stores have significant performance penalty compared to classical block/file access and, if you don't scale out, you lose many of their benefits.

That said, excluding classical RAID you have the following possibilities:

  • unraid, which uses a file-level replication approach rather than a block-level one

  • glusterfs, configuring a different brick for each disk and applying a replica 2 policy

  • zfs which, albeit block-based, is not 100% identical to classical software or hardware RAID

Also note that object storage does not guarantee lower rebuild time; quite the contrary, it tackles long rebuilds by guarantee stronger replication policies (ie: replica 3 over different racks).

shodanshok
  • 47,711
  • 7
  • 111
  • 180
  • The big upside of ZFS in my book is that it can actually tell you which file(s), if any, have been corrupted; and if you're able to read them, you know they are uncorrupted. (Except if you've done `zfs set checksum=off`, but I can't imagine why anyone would run ZFS and do that.) Even absent storage redundancy, this can be a big advantage because you'll know exactly which of your boss'... er, important spreadsheets... need to be restored from backup. – user May 19 '19 at 20:50
7

"that will ensure data safety but without the drawback of huge rebuild time when replacing a disk"

You have a delusion in that you think software can change reality - i.e. physics. The hugh rebuild times essentially run down to having to write a compelete Disc and if you write 2tb of info on a disc it takes a long time BECAUSE THAT IS HOW DISCS BEHAVE. Particualarly if you HAVE to read multiple discs to get the missing data in the first pleace (Raid 5,6 analog). There are details in how "stupid" software can be (i.e. add time on top of the physical minimum), but the minimum already is hours if you talk of slow large SATA discs.

All other technology you mention has to adhere to the same problematic large time you want to avoid because - physics.

And it does not help that your cluster network - if you do replication of the the network - is using 10g or lower (which is the most likely case) instead of going 40g or 100g.

TomTom
  • 51,649
  • 7
  • 54
  • 136
  • My employer makes a network analyzer which is essentially a standard PC with 2 * 100 GbE standard NICs that analyzes uncompressed video streaming traffic in realtime. So, it's not entirely outlandish, but pragmatically, yeah, network speeds are likely going to be an additional bottleneck. – Jörg W Mittag May 19 '19 at 12:18
  • Well, my servers all are 2x100gb nic these days - but then my main customer's IT department sells 1gb (!) backend network speed as "high performance". The world seems to run on prople that do not even know what high end is. – TomTom May 19 '19 at 13:06
  • 1
    Wouldn't the rebuild time be drastically reduced if all the disks were SSD? – Mike Waters May 19 '19 at 19:40
  • 1
    Depends. Ok, no - yes, it would be reduced, but SATA also tops at 600mb/s and if you run a hyperconvergent network that also means it must go over the network. And you do not even need all SSD - a decent SSD cache should also help with read and write (I use 3.6gb SSD cache on every server). But really you need a backbone to handle this, then. – TomTom May 19 '19 at 20:10
  • 2
    Mike said SSD, not SATA. There are some PCIe-connected SSDs that are faster than 600MB/s. – Peter Cordes May 19 '19 at 21:09
  • yes, physics show also that in some case there is differences : https://iq.quantum.com/exLink.asp?11054880OB98P19I42661089&TB00022A&view=1 – dominix May 20 '19 at 02:02
  • I know. If you run a proper server and have some U.2 slots available you get 4 full PCIe channels and a very low latency. But that is already high end, as is 100G ;) – TomTom May 20 '19 at 05:35