5

I am currently in the process of building a new storage server, to be used for virtual machines, files and backup. The OS is FreeNAS, which uses ZFS as software RAID.

My problem is that, I need to choose hard drives and I have looked at both consumer and enterprise hard drives, but i am faced with an unanswered question, which I have not been able to find a clear answer for.

Can I use consumer hard drives, that do not support TLER / ERC / CCTL, with ZFS (software RAID), without getting in trouble later on or do i need to go with enterprise hard drives that supports TLER / ERC / CCTL?

There are a lot of different opinions about this and some say you should use it and some say you should not. I know a couple of OS which relies on software raid, that have their own error correction timeout and therefore does not care if there is TLER / ERC / CCTL. I am aware, that you need to use TLER / ERC / CCTL, when dealing with hardware RAID.

I really hope, that someone can shed some light on my problem.

Thanks.

Indigo
  • 53
  • 5
  • From my reading you can go without on a ZFS system without a loss. I do hope you're taking into consideration the longer average life span for enterprise drives. If this is a business the minor price increase is totally worth not needing to worry as much. – Tim Brigham Mar 12 '17 at 15:06
  • Yes, i am aware of the longer life span and over all reliability on enterprise drives. This is a home project, so that is why i am a little picky, about the extra cost of enterprise drives. I am only going to use the consumer drives for non important bulk storage, but it would just be nice to know, that i will not be "shooting myself in the foot", when using consumer drives in a software RAID. Thanks for the answer. – Indigo Mar 12 '17 at 15:30
  • I think if you want to use RAID then you should use disks specially for RAID. Desktop version disk will trying to live when it is bad. It will trying to read bad sector many times, and when it will be read it the disk will give RAID this sector as good. RAID will get two sectors from good and bad disks with different data. This will stop RAID (hardware or software). Only you can assign bad disk as bad after this situation. I used desktop disks in RAID and sometimes my operating system went to the kernel panic state. If it isn't important for you then you can use desktop disks in RAID. – Mikhail Khirgiy Mar 13 '17 at 05:35

2 Answers2

2

Can I use consumer hard drives, that do not support TLER / ERC / CCTL, with ZFS (software RAID), without getting in trouble later on or do i need to go with enterprise hard drives that supports TLER / ERC / CCTL?

Imagine each of your drives is a black box with certain features, lifetime etc. All of them work independently inside your storage array. Therefore, you must look at each of them independently to see what would happen in different cases.

Example

For this question, let's say you have a pool consisting of 2 mirrored drives A and B. The possible combinations are:

  1. A and B have TLER
  2. A has it, B does not
  3. A does not have it, but B does
  4. A and B do not have it

If everything works fine on all disks, there is no problem.

One error:

If disk A experiences an error when trying to read a block, it is this situation for each possible case:

  1. The system asks the disk for the block. The disk tries to find it again and again, and after about 7 to 9 seconds (whatever the TLER timeout is) the controller drops the disk from the array. The system noticies that one disk is missing and does whatever you have provisioned (raise an email alert, begin resilvering of a hotspare, do nothing and continue degraded, ...)
  2. Same as 1.
  3. The system asks the disk for the block. The disk tries to find it again and again, until its own timeout is reached or until the block is found. This timeout may be several minutes and may be different for each different model and manufacturer. If the block can be retrieved, it is read normally; if it fails, an error message for the block itself is returned.
  4. Same as 3.

Two errors (one on each disk):

This case is very rare, but theoretically possible. Both disks experience an error on exactly the same block:

  1. Both disks will be dropped and your pool becomes unavailable.
  2. Same as 3 from above.
  3. Same as 3 from above.
  4. Same as 3 from above.

Performance vs reliability

As you have seen above, you have to make a choice depending on your goal and pool layout.

  • Use TLER disks if you always need minimal response time and cannot accept long blocking lag. The downside is you have to potentially provision additional disks (Z2 instead of Z1, Z3 instead of Z2, 3-mirrors instead of 2-mirrors) or hotspares to achieve the same average pool health in the end. Also you have to provision for potentially long rebuild times which may affect your overall performance negatively.
  • Use non-TLER disks if budget or space is constrained and lag is acceptable if disks are preserved. As ZFS already helps you with self-healing for affected blocks, you do not need what TLER was originally envisioned for (controller must drop disks so that self-healing could start)
  • Set the disk timeout to a value that your application/architecture can comfortably handle. This way you can have enterprise disks without dropping or consumer disks with dropping as you prefer. Not all disks are modifiable, so check online first before buying.
user121391
  • 2,502
  • 13
  • 31
  • Thank you, that was exactly the technical explanation i was looking for. It also confirmed my theory, to some degree, that ZFS could handle the disk errors, if a consumer drive array fails, but of course with some overall degration. – Indigo Mar 15 '17 at 12:45
  • @Indigo If you want to simulate the worst case on consumer disks, just mount an NFS share, shutdown the remote NFS server and view the directory... ;) – user121391 Mar 15 '17 at 12:50
1

Also worth considering are the figures Backblaze published about enterprise drive lifetimes, and consumer drive reliability.

Although they use a custom Reed-Solomon implemention, their figures (and business model) suggest that consumer drives are definitely able to provide good reliability, and with a suitable error detection/recovery algorithm, can provide good protection for your data. Certainly, their entire setup seems to have fared pretty well without any of these enterprise features.

So as @user121391 says (I may be paraphrasing a little), ZFS on consumer disks should be fine, unless you have specific needs that would require enterprise features.

iwaseatenbyagrue
  • 3,688
  • 15
  • 24