5

I want to set up a RAID 5 with 3x1TB drives on a Linux KVM-Host. The RAID will be used as LVM thin storage for VM disks.

There has already been much discussion on hardware raid vs software [1]. According to this discussion one should not use software raid but hardware raid with cache and BBU for VM disk storage because of the better write performance.

What I would like to know is if the following setup would be comparable to a hardware raid with cache and BBU (e.g. HP P410 512 MB + BBU) in terms of read/write performance and data safety:

  • Linux Software RAID / mdadm RAID 5
  • LVM writeback cache on a 512 MB ram disk
  • Host backed by UPS to prevent data loss like the BBU on hw raid

[1] Software vs hardware RAID performance and cache usage

c0d3z3r0
  • 53
  • 5

2 Answers2

9

None from above! You really need to look at ZFS on Linux.

http://zfsonlinux.org

https://www.reddit.com/r/zfs/comments/514k2r/kvm_zfs_best_practices/

Perfect discussion, tons of links.

BaronSamedi1958
  • 13,676
  • 1
  • 21
  • 53
  • 1
    Yeah, I've had a look at ZFS but I don't want to use third-party kernel patches or modules and this doesn't really answer my question. Thanks anyway. – c0d3z3r0 Jul 02 '17 at 18:55
  • 2
    The problem is with your config - 5x1TB spinners in RAID5 you're getting to a point where limit of a single parity RAIDs really kicks in. I mean both hardware RAID and LVM are dangerous in terms of a slow rebuilds. ZFS with RAIDZ1 is close to RAID5 but a) rebuilds are faster so less changes to hit double fault and lose all your data, b) there are hashes so silent bit rot isn't going to happen, and c) there's no write hole as ZFS has variable size parity strips. TL;DR: Don't do hardware or software parity RAID with 1TB SATA drives. – BaronSamedi1958 Jul 02 '17 at 19:19
  • Hmm, maybe I should really simply try ZFS... Thank you for explaining the details :-) – c0d3z3r0 Jul 03 '17 at 17:34
  • Best of luck! :) – BaronSamedi1958 Jul 03 '17 at 20:14
0

This is an old but still-relevant question.

Linux Software RAID / mdadm RAID 5

A "plain" MD RAID5 of HDDs or consumer SSDs will be slow. To get good performance, one has to use powerloss-protected writeback cache SSDs or use a RAID journal device (itself on a fast, powerloss-protected SSD).

LVM writeback cache on a 512 MB ram disk

Writeback on a RAM DISK is going to be a bad idea: in case of powerloss, you will completely corrupt your volumes. It is way better (still dangerous) to simply disable barriers / sync writes putting the kernel in "write through" mode (see here the description about write_cache). A power loss will cause data loss, but hopefully not the cataclysm that losing the entire writeback device would be.

Host backed by UPS to prevent data loss like the BBU on hw raid

Powerloss will happens. An UPS is a first line of defense, but not let it be the only one.

What you can do:

  • use lvmcache to use a fast, enterprise-grade, powerloss-protected SSD to cache your slow pool. If using a single SSD select LVM writethrough, while if using an SSD RAID1 pair you can select LVM writeback (be sure to understand what you are doing, though). Note: you probably want to use LVM cache rather than LVM writecache (the latter is for a more specific use case);
  • use ZFS with L2ARC and SLOG devices. This is my recommended setup: you can use read-centric, fast but inexpensive devices for L2ARC, and a more pricey, powerloss protected SSD for SLOG (a mirrored pair of such SLOG devices is a good idea for maximum reliability).

For what it is worth, I run many systems with the above ZFS setup and I am very happy about their performance, reliability and data-correctness guarantees.

shodanshok
  • 47,711
  • 7
  • 111
  • 180