1

I am building fast recovery solution. And there is a stuck with the choosing of the RAID controller.

I have a chain:

  • 24 SATA SSD drives each performing at 4Gbit/s. Total write throughput is 96Gbit/s.
  • SAS Expander Intel RES3TV360. 2x4 SAS IN. 7x4 SAS/SATA OUT. 12/6/3/1.5 Gbit/s per port.
  • RAID Controller Intel RS3P4MF088F. 1x8 SAS/SATA OUT. 12/6/3/1.5 Gbit/s per port.

In both manuals for RAID Controller and SAS Expander stated:

  • Up to 12Gbit/s per SAS port
  • Up to 6Gbit/s per SATA port
  • STP (SATA tunneling) via SAS ports supported

Drives are organized in RAID 10 array.

The question is: Which top write speed I can expect and where is a bottle neck in this chain?

I can explain my concern:

Neither in RAID Controller nor in the SAS Expander guidelines stated at which speed RAID Controller and SAS Expander will communicate via 8x lines between each other when only SATA drives are connected to SAS Expander.

  • If it will be 12Gbit/s then 8 x 12Gbit/s = 96Gbit/s - and it is fine. (It results in 48Gbit/s write speed on RAID 10 - and it is acceptable)

  • If it will be just 6Gbit/s between RAID Controller and SAS Expander then 8 x 6Gbit/s = 48Gbit/s - and it is not enough. (It results in just 24Gbit/s write speed on RAID 10 - and it is less than expected for my solution)

Thanks in advance.

Mitchel
  • 33
  • 5

2 Answers2

2

With rare exceptions, expander cannot change the transfer speed between host and disk, so 6 Gbit/s per lane is the best you can get with SATA. SATA switching is also less efficient than for SAS, so you should expect a significant reduction in actual speed (I'd estimate 3-4 Gbit/s per lane).

Zac67
  • 10,320
  • 2
  • 12
  • 32
  • It is clear that I cannot expect more than 6Gbit/s per lane between SAS Expander and SATA drive and it is fine. The question is about bandwidth between RAID Controller and SAS Expander. Will it be 6Gbit/s or 12Gbit/s per line in between if SATA drives are connected to SAS Expander? E.g. if I have 16 SATA drives performing 6Gbit/s connected via SAS expander to RAID Controlled. And RAID Controller and SAS Expander are connected with 8 lines. What will be max throughput between drives and RAID controller: 6G x 8 lines = 48Gbit/s or 12G x 8 lines = 96Gbit/s – Mitchel Jul 08 '22 at 13:05
  • Without changing the transfer speed on the expander (which is very rarely done), the entire path is limited to 6 Gbit/s with SATA drives. So, 6G x 8 = 48G. – Zac67 Jul 08 '22 at 13:37
  • As I understood from number of explanations SSP (scsi) and STP (sata/ata) traffic flows from Controller to Expander in common way with the top supported speed (12G). All Expanders doing is routing of packets. Packets addressing is implemented in the top level of SAS packets so switching of SSP and STP packets the same. Expander also switches target port to SAS or SATA mode according to target device/drive type. Seems at this point speed falls down to 6Gbit/s for SATA drives. Another words if I have 2x12G lanes between Controller and Expander, I can get 4x drives performing at 6G. Am I correct? – Mitchel Jul 08 '22 at 13:45
  • Seems I found what you a talking about. In the original SAS spec. it is stated that Expander always sends packets synchronously. In case if low-speed device is connected to the high-speed expander which is connected to high-speed controller, then expander adds ALIGN frames, so that link between controller and expander can be high speed, but still synchronous. So seems this issue relates any array having devices/drives of different connection speed (not only SATA). Solution is in using extenders, which caches data from number of low-speed devices and transfers it via single lane. Like LSI SAS3. – Mitchel Jul 08 '22 at 15:42
  • Yes, exactly! This approach solves the problem of buffering and possible out-of-order delivery. Buffering and asynchronous delivery requires significantly more logic (and budget). For you it's likely cheaper to use 12G SAS SSDs. – Zac67 Jul 08 '22 at 16:15
  • I'd be happy to Use SAS drives. But I have a deal with pretty big amount of data. Even SATA drives x24 will cost me $12-14k per server. SAS drives of such volume will cost few times more and it will be 96 drives solution, because there are no SAS drives of needed capacity. In my conditions selection of proper controller is critical task and I can spend $1-1.5k more per controller if needed. In the Intel expander docs it is stated "Supports multiple data rates and auto-negotiation between following: 1. SAS 3/6/12G 2. SATA 2/6G" But there no any info how expander performs with SATA drives. – Mitchel Jul 09 '22 at 19:56
2

The answer is 96Gbit/s total throughput and 48Gbit/s hypothetical write speed for RAID 10 in short. Described chain will perform on expected 12Gbit/s per lane between RAID Controller and SAS Expander. But it is not an universal rule for all the SAS Expanders!

Details:

  • Initial SAS protocol is designed to work synchronously - directly transfer the data. No any storing of data. Just routing basing on address in the very beginning of the data frame, and then direct non-stop transferring of the data frame content. It works fine if all the devices can perform at the same speed e.g. 12G. When you connecting low-speed (for example 3G) device, typical SAS Expanders continues to work synchronously with all the devices. If some end-device performs slowly, SAS Expander though transfers data synchronously by adding empty ALIGN frames (in case of 12G link between RAID Controller and SAS Expander it will be 3 ALIGN frames per 1 data frame). It is so called "rate matching" feature of the SAS protocol. Another words it occupies whole 12G bandwidth lane between SAS Expander and RAID Controller transferring just 3G of data.
  • Such solution without store-forward approach was implemented in order to provide lightning-speed real-time performance and avoid any delays, which could be up to 1ms for each input/output operation, having technologies of the time of SAS 1.0.
  • Connecting SATA drives (which are max 6G) to simple 12G SAS Expanders results in reducing of performance equal to lowest speed device in the SAS net.
  • But many of smart modern mid- and high- price level SAS Expanders supporting SAS3 are designed to work in Store-Forward mode with data frames coming from SATA drives. They performs caching of the data frames coming at 6G from SATA and transfers them at 12G between SAS Expander and RAID Controller, utilizing full speed of 12G SAS lanes. Keyword to detect such SAS Expanders are "SAS and SATA edge-buffering support" (Microchip) or "SATA data buffering" (Broadcom). See patent for explanations https://patents.google.com/patent/US8924610B1/en
  • SAS Expander Intel RES3TV360 (https://www.intel.com/content/dam/support/us/en/documents/server-products/raid-products/res3tv360_ug_002.pdf) selected for the solution is based on Microchip PM8044 (https://www.microchip.com/content/dam/mchp/documents/DCS/ProductDocuments/Brochures/00003036A.pdf), which supports "SAS and SATA edge-buffering".

Thus we can assume that this solution having 8 x 12G lanes between RAID Controller and SAS Expander will operate with SATA drives array at 96G providing theoretical 48G write throughput in case of RAID10 as expected.

Special thanks for comments provided!

Mitchel
  • 33
  • 5