Building/Maintaining a Custom FreeNAS- and ZFS-based NAS

Question

I need a NAS for a company of ~30 people. We make games (a lot of large files, large git repositories, not much else.) I'm thinking of buying cheap components and building it myself, using something like FreeNAS. (Assume that it is a viable option, price-wise.)

I'll list my needs and concerns below, but my main question is this: is it easy enough to build and administer, or should I just buy a commercial NAS (I have used Synology and it fits all my needs.)

My needs:

1. Data integrity is paramount (obviously!) but I don't need 100% uptime. If the CPU dies, I can take 2 hours to replace it.

2. It should not need constant tinkering and maintenance. I want to stick the box in a corner somewhere and just log into it once a month or so when I want to add a repo or user. That being said, I can tolerate a moderate amount of work and complexity to set up the box, or a new service; but after that I want it to "Just Work".

3. I want to start off with a bunch of different sized hard disks (basically what we have lying around,) and continue to upgrade them one by one with disks of different sizes as space runs low or they die. It is unacceptable for me to have to buy 8x4TB disks today and then replace them all at the same time with 8x10TB disks when the array runs out of space in two years. (Synology's Hybrid RAID does a great job here; a feature which I am yet to find in other commercial NAS boxes.)

4. Should have a graphical user interface (ideally, web-based or something) for common tasks for non-technical users.

5. The most important services I'll be running on this box will be Samba shares, and Git repositories. I need access control (nothing too fancy or complex) and good network support.

My concerns:

1. Is FreeNAS the right OS for this? I can run a Linux distro and set everything I want up using software RAID and LVM and whatnot, but that is a little out of my acceptable complexity range (or right at its edge,) and I prefer something simpler and more straightforward for the job.

2. If FreeNAS is the way to go, then I'll be worried about it being based on FreeBSD, with which I have no experience. Is it really too different from Linux? (I'm quite comfortable in almost any Linux terminal.) I have little problem with building software from source, but a good package manager (a la apt, etc.) is a huge plus. Should I expect any problems in hardware support? (basically the only special hardware that I can think of will be the NICs.)

3. Is ZFS the way to go? From what I read, ZFS (and RAIDZ) is pretty awesome and basically what I've been looking for my whole life! Is it? What are the gotchas and the most important things I have to keep in mind, coming from basic hardware RAID/LVM, and ext4/NTFS world?

4. I'm thinking that a cheap 2- or 4-core CPU, and ~4GB of RAM will be enough to run this. Is it? This is hardly critical, but I'd like to know if I'm way off.

5. Network link aggregation (NIC bonding) is pretty painless in Linux. I'm reading that FreeBSD is the same. Is it?

6. I'm going to be putting several terabytes of mostly irreplaceable data into this box; data that will be very hard to move if this doesn't work out. What else should I research, think about, or keep in mind?

I know I'm basically asking the community for consultation! But I'd really appreciate any and all help. Thank you.

UPDATE: One important question I forgot: in such a system, what would the process of replacing a hard disk be like? (either because of failure, or an upgrade.) Is it too hard? (please excuse the too obvious pun!)

UPDATE 2: Based on the answers below, and other "research," I've decided on the following strategy. I'm writing it here on the off-chance that it would be useful to someone else in my situation. (This is not a final decision; but rather a procedure for further investigation.)

I'm going to buy the hardware components (I'm going to use cheap stuff, since the only load will be I/O, and limited by a GbE pair) and with the free disks we have lying around, build a FreeNAS-based appliance (although with more RAM than I originally intended.) I'll build the storage array, and before putting any real data on it, will play around with it for a bit (We're in no rush.)

If the underlying FreeBSD proves too much of an alien beast, I'll probably switch the same test system to Linux and ZFS (and other software mentioned in the accepted answer below.)

If that doesn't satisfy either, I'll definitely give unRAID a try.

If none of the above work out, I'll just re-purpose the box for something else, and buy an off-the-shelf NAS (probably Synology 1817 or something like that.)

At any point, if I'm sure that I can maintain and serve my data on/from any of the above solutions, I'll start moving our actual stuff over.

Answer 1

Just want to address a few of your points. Full disclosure, I am a proponent of ZFS. This is all IMHO.

RE: your needs:

2. It should not need constant tinkering and maintenance.

As long as all your scrubs, snapshots, security updates, capacity alerts, etc. are configured correctly, it should just work (at least until it phones home). Buggy software (looking at you netatalk) notwithstanding.

3. I want to start off with a bunch of different sized hard disks...

As willcoq pointed out, ZFS is not great on this particular point. At a bare minimum you'll need pairs of identical (or nearly-identical) drives to mirror (which will give you 50% storage efficiency). This also happens to be the most commonly-recommended configuration.

...and then replace them all at the same time...

And as I pointed out to willcoq, if you go with mirrored-pair vdevs, you can upgrade your pool two disks at a time. If you go with two four-disk RAIDZ2 vdevs, you can upgrade your pool four disks at a time. If you go with a single seven-disk RAIDZ3 vdev (plus a hot spare), yes, you'd need to replace all the drives to realize the expansion. Et cetera.

...Synology's Hybrid RAID does a great job here; a feature which I am yet to find in other commercial NAS boxes.

Hybrid RAID (Synology), BeyondRAID (Drobo), XFS (e.g. unRAID), ZFS (e.g. FreeNAS), etc. all come with trade-offs. If being able to throw random drives in a thingy and have it do magic is your primary requirement, that would definitely push you toward one of the first two options.

Should have a graphical user interface (ideally, web-based or something) for common tasks for non-technical users.

FreeNAS shines here compared to other ZFS-based solutions.

RE: your concerns:

3. Is ZFS the way to go?

ZFS doesn't necessarily have a steeper learning curve than, say, rolling your own XFS solution, but the architecture is perhaps less forgiving; it's possible to make critical, difficult-to-fix errors in the design phase. Unfortunately, there's a lot of "tribal knowledge" and there's no way around reading and learning as much as you can while planning your solution. ZFS-based solutions definitely have a steeper learning curve than commercial, off-the-shelf systems from Synology and Drobo, and commercial software like unRAID.

With that complexity, however, comes power. You can build exactly the system you need for your use case, with few or no compromises.

2. If FreeNAS is the way to go, then I'll be worried about it being based on FreeBSD, with which I have no experience. Is it really too different from Linux?

I was in a similar boat and definitely had difficulty learning FreeNAS, FreeBSD, and ZFS all at the same time. There was something about the presence of a GUI that blocked me, mentally, from diving into the command-line—if I couldn't figure out how to do it in the GUI, I started to wonder if what I was trying to do was worth doing—or even possible. I've built my last few storage "appliances" using ZFS-on-Linux and it's been a better fit for me. It has forced me to learn ZFS and it lets me run Docker and LXC/LXD on ZFS without a VM layer. Unfortunately, it's not as user-friendly for the less technically-inclined among us.

If you're thinking about going the ZoL route, you should take a look at Proxmox, which is essentially Ubuntu + ZoL + ZFS root + LXC/LXD + libvirtd + GUI.

1. Is FreeNAS the right OS for this?

If you decide to go with ZFS and need a GUI on day one, the short answer is yes, probably. It might be worth trying to learn FreeBSD and ZFS fundamentals separately before jumping into FreeNAS.

4. I'm thinking that a cheap 2- or 4-core CPU, and ~4GB of RAM will be enough to run this. Is it? This is hardly critical, but I'd like to know if I'm way off.

A cheap-ish CPU should be fine as long as it has AES-NI (for encryption). If you want to run other applications on this server, you may want to consider additional sockets/cores.

The rule-of-thumb for ZFS is one gigabyte RAM per terabyte (usable) storage, plus whatever you need for your operating system and applications. For better read performance, add additional RAM. ZFS will try to take 50% of your RAM by default; this is tunable, to a certain extent (base-2 values).

5. Network link aggregation (NIC bonding) is pretty painless in Linux. I'm reading that FreeBSD is the same. Is it?

Yup. Just be sure to do a bit of research about which NICs work best in both. Hint: not Realtek.

6. I'm going to be putting several terabytes of mostly irreplaceable data into this box; data that will be very hard to move if this doesn't work out. What else should I research, think about, or keep in mind?

Redundancy is not backups. You should be thinking of backup strategies for your critical datasets.

Answer 2

I’ve never used FreeNAS without also using ZFS, and ZFS is NOT friendly to number 3 on your needs list - being able to swap out different sized disks one at a time. You can do it, but you won’t be able to utilize the extra space until you’ve replaced ALL the disks in the ZFS pool. You can have 8x4TB disks today, and you can swap out each 4TB with a 10TB one at a time, but the extra 6TB on each of those new disks won’t be utilized until you’ve replaced all 8.

That being said, FreeNAS might have non-ZFS options, but I’ve never explored them.

I think StableBit’s DrivePool can do what you want with the drives, but you can’t use FreeNAS for that.