1

I'm thinking of building a ZFS server optimized for random access (PostgreSQL database). I figured that I have 2 options:

  1. Use pools consisting of the Crucial M500 960 GB SATA III SSDs. Enterprise SAS SSDs are not price-effective, considering that they could cost 10x as much as their SATA III counterparts.

  2. Use pools consisting of 15K SAS HDDs. (In contrast, the price differential between an enterprise SAS HDD and a regular SATA III HDD is much smaller, so buying a SAS HDD is still justifiable.)

My questions are:

  1. SATA III SSD vs 15k SAS HDD: Most of the arguments against SATA III have to do with the duty cycle (not suitable for 24/7?), but given that the MTBFs of consumer SATA III SSDs (e.g. Samsung 840 Pro, Crucial M500) are comparable to those of SAS SSDs and 15k HDDs, is there anything wrong with using the M500s? Or am I better off with the 15k SAS HDDs despite the clear performance advantages of the SSDs for random access?

  2. L2ARC and SSD ZFS pools: If I'm using SSDs-only for my ZFS pools, would I benefit from an L2ARC device?

elleciel
  • 389
  • 4
  • 11

1 Answers1

3

My recommendations vary a bit based on the distribution and approach to ZFS...

  • SSD-only pools work well - However, you need to think very carefully about the controller, backplane and drive layout. Main considerations are: no expanders, use pure SAS HBAs, ZFS mirrors are king.

  • 15k SAS disks in 2014 are Meh - If you need the random I/O performance of SSDs. You can used SAS disks as another tier or even nearline drives if you need the capacity.

  • L2ARC won't be necessary for your all-SSD pool - Optimize and maximize RAM. There are some considerations for Linux-based ZFS solutions, but in general, more RAM is better than gobs of L2ARC, especially if you don't know your I/O pattern and working set.

  • Miscellaneous - Use compression, monitor and scrub your pool, use good RAM, test, buy quality SSDs... Crucial isn't on my radar. There are other high quality SATA SSDs available that will have better performance characteristics.

ewwhite
  • 197,159
  • 92
  • 443
  • 809
  • I've heard that the algo prioritizes most *frequently*-used data set for L2ARC and most *recently*-used data set for the ARC, so assuming my most recently and most frequently-used data sets are different, there's benefit to having an L2ARC cache even if I had a huge amount of RAM - is this correct? – elleciel Feb 18 '14 at 23:53
  • 1
    @elleciel Depends on working set size and distro. My recommendation for Solaris-derivatives is different than Linux... For the former, I'd use L2ARC. For the latter... I've been doing some cool stuff with ZRAM and L2ARC. A lot of things need to be set in order for ZFS to behave on Linux, so I can only give a general answer now. Also, your SSD matters. For a high-quality low-latency SAS SSD, I would be okay going to the pool most of the time. – ewwhite Feb 18 '14 at 23:58
  • Also, let me get your first point correct (backplane and drive layout). I should (1) have the disks placed internally, i.e. in the same chassis as the head node, _instead_ of placing them behind an SAS backplane on an external JBOD chassis, and (2) connect my SATA drives directly to pure SAS controllers using SFF-8087-to-4x SATA cables _instead_ of using SAS-SATA interposers? – elleciel Feb 19 '14 at 00:01
  • I see, that's helpful... I'll almost certainly be using SATA SSDs as SAS SSDs are too pricey. – elleciel Feb 19 '14 at 00:05
  • 1
    @elleciel Depends on the chassis, the server backplane and the disks. If you were to say Supermicro was in use for JBODs, I'd say "stick with SAS". If you were to say an HP JBOD was in use, you could do SATA. In either case, you lose SAS multipath... so there are some design considerations there, too. – ewwhite Feb 19 '14 at 00:06
  • 1
    A best case is a server with 1:1 controller lane to slot mappings... So how many disks are you planning? How much capacity do you need and what hardware are you planning to use? – ewwhite Feb 19 '14 at 00:08
  • Can't thank you enough. My original plan was 2 RAIDZ-3 zpools of 11 disks each. Each disk would be a 960 GB SATA SSD. We use Supermicro exclusively... Exact case and mobo would depend on the other design factors but I was tentatively planning to use a Supermicro 216BA-R920WB. I'm guessing this rules out the SATA SSDs and I'm forced by budget to go with the 15k SAS disks... :/ – elleciel Feb 19 '14 at 00:14
  • 1
    @elleciel you can use Supermicro... But look here for more best-practices: http://nex7.blogspot.com/2013/03/readme1st.html – ewwhite Feb 19 '14 at 00:25
  • Gotcha, thanks for the link. I'll go back and do some homework based on your recommendations. – elleciel Feb 19 '14 at 00:38
  • In general, RAIDZ3 in the config you're suggesting will be very slow. – ewwhite Feb 19 '14 at 00:39
  • P.S.: I mixed up vdev and zpool in my above posts, I meant to say 2 RAIDZ-3 vdevs of 11 disks each. Hm, could you elaborate? Should I drop it to 7 disks per vdev? I'm planning another build with 4 TB nearline SAS hard drives, 11 drives per vdev as well, and that would perform even worse if that's what you're saying. In the case of the 4 TB drives, I figured triple parity would be necessary given the value of the data and the time it takes to resilver the drives. – elleciel Feb 19 '14 at 06:30
  • 1
    @elleciel ZFS scales across vdevs. Your proposed design would have the random IOPS performance equivalent to *two* disks. Use mirrors when and if you can. – ewwhite Feb 19 '14 at 08:11
  • Got it, I've a better idea how it works now. What about something like 6 RAIDZ-3 vdevs of 7 disks each? Trumps 14 3-way mirrors in capacity, better redundancy, still cheaper (capacity/$) than a hardware RAID-10 build. – elleciel Feb 19 '14 at 21:48