11

When deciding whether to use LVM volume groups or ZFS pools in configuring new file servers, what needs to be considered? Is there a "better" choice in the realm of multi-purpose file servers? Unlike this previous question, I don't want to layer the technologies.

Scenario:

  • RHEL / CentOS 6 x64 servers
  • many available, identical DAS and SAN LUNs

Choice:

I am personally quite familiar with LVM, so am comfortable using it if it is the better option. However, ZFS looks pretty promising, and learning new technology is always good.

Given that we want to be able to share-out a fairly large store (multiple TB) to different departments, and they need to be able to access them over both CIFS and NFS, should we use ZFS or LVM on for the underlying pool manager?

I know that using a product like FreeNAS is possible, but for a variety of reasons, I need to be able to roll-out onto "typical" Linux servers.

Community
  • 1
warren
  • 18,369
  • 23
  • 84
  • 135
  • 5
    In short, ZFS filesystems are the way to go... I'll follow up later. – ewwhite Apr 17 '14 at 16:27
  • 4
    I've been using ZFS for a few years now. The most compelling argument in its' favor (in my opinion) is [this](http://4.bp.blogspot.com/_mEl5xoSoWPw/S3x_V2Ry3RI/AAAAAAAAAHA/ysRKONNsDq0/s400/MTTDL-1-12disks.jpg) (from [here](http://blog.richardelling.com/2010/02/zfs-data-protection-comparison.html)) showing mean time to data loss. Anyway, that's why I am running raid-z3. – Elliott Frisch Apr 17 '14 at 16:42
  • Regarding FreeNAS: this might be a nice solution, but as many NAS-like solutions, there are quite a few things hidden (ie a small swap partition at the beginning of each disks, ...). You might learn a thing or two playing with the beast, but if you know what you are doing, then you should do it all by yourself. – Ouki Apr 17 '14 at 23:39
  • @Ouki - in another context, we will be deploying FreeNAS, or something similar; in this context we aren't able to :) – warren Apr 18 '14 at 14:05
  • @STW - while I appreciate your opinion, I have to wonder why you are so vocal in disagreeing with such an established member of the SF community :) – warren Nov 26 '14 at 21:37

1 Answers1

13

I use both, but prefer ZFS. ZFS on Linux has been very good to me, but isn't the "fix all" for every situation.

A typical server will look like this:

(Remember, I usually use hardware RAID and mostly use ZFS as a flexible volume manager)

  • Hardware RAID with a logical volume comprised of underlying disks. That array will be carved into a small OS volume (presented as a block device), then partitioned (/,/usr,/var and such).
  • The remaining space will present another block device to be used as a ZFS zpool. 
Smart Array P420i in Slot 0 (Embedded)    (sn: 001438029619AC0)

   array A (SAS, Unused Space: 1238353  MB)

      logicaldrive 1 (72.0 GB, RAID 1+0, OK)
      logicaldrive 2 (800.0 GB, RAID 1+0, OK)

      physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SAS, 900.1 GB, OK)
      physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SAS, 900.1 GB, OK)
      physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SAS, 900.1 GB, OK)
      physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SAS, 900.1 GB, OK)

I then take the zpool and create the additional ZFS filesystems (or mountpoints) and zvols as necessary.

# zpool list -v vol1
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
vol1   796G   245G   551G    30%  1.00x  ONLINE  -
  wwn-0x600508b1001c4d9ea960806c1becebeb   796G   245G   551G         -

And the filesystems...

# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol1                245G   539G   136K  /vol1
vol1/images         245G   539G   245G  /images
vol1/mdmarra        100G   539G   100G  /growlr_pix

So, using ZFS for data partitions is extremely nice because it allows you to address a pool of data, set quotas and manage attributes at mountpoint granularity. LVM still requires dealing with filesystem tools and is a bit more rigid.

ewwhite
  • 197,159
  • 92
  • 443
  • 809
  • so - are there *any* pros to LVM over ZFS in the Linux space? – warren Apr 18 '14 at 14:05
  • 1
    @warren LVM has mindshare. I'm not terribly good with it, but many Linux administrators are very familiar with LVM. So, from a support and knowledge-transfer perspective, there's an argument for it. – ewwhite Apr 18 '14 at 14:09
  • 7
    using ZFS on RAID is discouraged and partly defeats the benefits of ZFS: http://en.wikipedia.org/wiki/ZFS#ZFS_and_hardware_RAID – STW Nov 26 '14 at 19:07
  • 1
    @stw there are use cases where it does make sense. – ewwhite Nov 26 '14 at 19:10
  • 9
    Your example above appears to completely undermine the benefits of ZFS. Having four physical disks as a single RAID logical disk in a zpool means that you don't gain ZFS's data integrity, your RAID controller will prevent ZFS from monitoring and responding to disk health, and you're sacrificing memory for effectively no gains. Your setup is an ideal candidate for LVM--not ZFS. Quotas, attributes (not supported by ZFS on Linux), and features such as compression can be had without ZFS. – STW Nov 26 '14 at 19:56
  • 2
    @STW you're not in a position to inform me of what works or doesn't in my environments. Take it somewhere else. – ewwhite Nov 26 '14 at 21:55
  • 5
    @ewwhite I would just note that anyone using this setup is risking catastrophic data loss. If you really only want to use ZFS for volume management that's fine, but it's only a small part of ZFS's functionality and loses the safeguards. – STW Nov 26 '14 at 22:25
  • 1
    @stw What the *eff* are you talking about? It's hardware RAID. – ewwhite Nov 26 '14 at 22:26
  • 7
    You're relatively safe if you're running RAID10 and have your zpool configured to `copies=2`. However in the event of a disk or controller failure you're more likely to end up with a complete loss of your pool than if you had the equivalent setup purely in ZFS. – STW Nov 26 '14 at 22:48