3

We have done a lot of research and got conflicting answers from experts. What is your opinion on the following?

  • 14 x 16 TB HDDs (Raid 10 -> 100 TB usable)
  • HW: SGL 8405 + AEC-82885T + BBU (replacement kept in stock)
  • 16 thread CPU & 256 GB ECC Memory

We need to build our own replicated S3 (minio) setup for big data cold storage. We spoke with experts on the matter of hardware vs software raid and got very conflicting answers. None could give strong explanations.

What is the state nowadays with HW vs madam raids? Is it still worth it to invest in HW for the main concern of reliability?

Any tips on how to pick which way to go?

user2693017
  • 203
  • 3
  • 12
  • When using a hardware raid, the system hardware might be more expensive, but managing such system as an admin is more easier and thus you are saving money in workforce. For software raid you also need a performant CPU and additional system memory for good performance. Another software raid technique is provided by btrfs filesystem. I would give it a try, it's a bit different to mdadm but has many advantages. PS do not use raid5/6 modes of btrfs, as those are experimental and unstable. But raid1 and raid10 are stable. – paladin Apr 26 '22 at 16:18
  • PS When using big raids you should usually go for scsi and when doing so, good scsi controllers are usually raid controllers. So going for hardware raid is usually the way to go. On the other hand, software raid systems allow more customization. – paladin Apr 26 '22 at 16:28
  • @paladin can you explain why? – user2693017 Apr 26 '22 at 17:06
  • @paladin what do you mean with maintenance time? madam is super simple and easy to maintain in my experience. We use it everywhere where we have SSDs or NVMEs. – user2693017 Apr 26 '22 at 17:07
  • An hardware raid is OS independent for example. OS independence also means that such system is more bullet proof vs cyber criminal attacks (at least when you can trust the manufacturer). Setting up a software raid takes more time and more maintenance than setting up a hardware raid (I'm also talking about installing the OS, configuring it proper and all the etc.). Hardware raid is usually faster when rebuilding the raid, hardware raids are also usually faster than software raids, especially with many drives. - This all saves time and money. – paladin Apr 26 '22 at 17:28
  • I see. It is not used as boot drive and if it would a preconfigured install image would take care of it all ;) Thank you for explaining. – user2693017 Apr 26 '22 at 17:30
  • PPS when using software raid, don't use mdraid, as it's outdated for big arrays. Better use a filesystem with implemented raid function, like ZFS or BTRFS. As those extraordinary reduce rebuild time and are easier to backup. (Especially BTRFS is very easy to backup.) – paladin Apr 26 '22 at 17:43
  • Thank you, @paladin – user2693017 Apr 26 '22 at 21:35

1 Answers1

1

Short answer: I would use ZFS (with striped mirrors) and call it a day.

Alternative answer: if you really want a traditional hardware RAID setup, go with some modern LSI-based cards as the ones based on Broadcom/LSI SAS3516 or SAS3916.

Be aware that AEC-82885T seems a plain SAS expander card, with no RAID or BBU (as a side note, it should be a good card for a ZFS SAS setup).

shodanshok
  • 47,711
  • 7
  • 111
  • 180
  • Thank you. The data that is stored is already compressed and de-duplicated. Therefore ext4 seems to be the better pick. Unfortunately only AEC-82885T is possible and yes the BBU is additional (not part of the card). – user2693017 Apr 26 '22 at 17:24
  • I forgot, SGL 8405 is the other part. – user2693017 Apr 26 '22 at 17:26
  • Adaptec SGL 8405 is a completely different card, a real HW RAID controller. Do you want to connect the SGL card to the 82885T one? Anyway, with such amount of data, I would reiterate that ZFS is the correct choice (compression and dedup apart). – shodanshok Apr 27 '22 at 07:49
  • I don't know how they are being connected but that is what the data center is offering/using as the only option (these two combined). So I suppose ZFS would have faster rebuild times than the HW with those disks? – user2693017 Apr 27 '22 at 11:17