3ware 9650SE: This Spare unit may replace failed drive of same interface type only

Question

I just added a spare disk to my array:

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-6    OK             -       -       256K    5587.9    RiW    ON     
u1    SPARE     OK             -       -       -       1863.01   -      OFF    

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.82 TB   SATA  0   -            ST32000542AS        
p1    OK             u0   1.82 TB   SATA  1   -            ST32000542AS        
p2    OK             u0   1.82 TB   SATA  2   -            ST32000542AS        
p3    OK             u0   1.82 TB   SATA  3   -            ST32000542AS        
p4    OK             u0   1.82 TB   SATA  4   -            ST32000542AS        
p5    OK             u1   1.82 TB   SATA  5   -            WDC WD2002FYPS-02W3

When I did, tw_cli said this:

Creating new unit on controller /c0 ... Done. The new unit is /c0/u1.
WARNING: This Spare unit may replace failed drive of same interface type only.

What does that warning mean?

Recently, I added a spare to another almost identical machine (with 9650SE as well) with no problems. The only difference was that in the other machine, the array was made up of WDC WD2002FYPS-02W3 disks, to which one was added. In this array, the array was made up of ST32000542AS disks to which a WDC WD2002FYPS-02W3 was added.

It's warning about the interface type, but all are SATA, so I don't get it.

Could this have something to do with the fact that the WDC WD2002FYPS-02W3 is a 4k sector drive and the ST32000542AS is not?

Which brings up a side-question: how does the 9650 SE align it's data? Is it compatible with 4k? Or maybe it's only 4k-aligned when it detects 4k sector disks (in which case, adding a 4k drive as spare will present a problem).

Answer 1

The warning about interface types can be ignored for your hardware, because 9650SE is a SATA-only controller. However, the same tw_cli utility can be used to manage newer cards like 9690SA which support both SATA and SAS drives, but mixing SATA and SAS in the same array is not allowed.

WDC WD2002FYPS does not seem to use 4k sectors — at least the specification sheet does not mention ”Advanced Format” anywhere, so there should not be any alignment issues with it. This drive is even included in the official compatibility list for 9650SE (but even though KB 15573 says that limiting speed to 1.5 Gbps is needed only for 9690SA, the actual compatibility list for 9650SE says “Only compatible if link speed set to 1.5 Gb/sec; see KB 15573”).

3ware 9xxx series controllers place DCB (array configuration data) at the end of each drive, and the user data is written starting from LBA 0 (e.g., see this and this posts), so there should not be any alignment issues from the controller itself.

But there is an alignment issue which is present even with 512-byte-sector disks (tested on 9690SA, not sure if 9650SE has the same problem). When you create multiple volumes from a single array either with autocarving (/cx set autocarve=on, or the equivalent option in BIOS) or with the “Boot Volume” option in BIOS, the specified volume size will be decreased by 1 (512-byte) sector, and all volumes except the first will be unaligned with respect to underlying RAID stripes. This can be observed in the tw_cli /cx show diag output — the controller log includes starting sector numbers for volumes in hex form, which should normally end with 00000 (1 GiB is 0x200000 sectors), but in the bad case they end with FFFFF, FFFFE, …, so you will see something like:

LUN 0x0 Start LBA 0x0 Size 0xf9ffffff
LUN 0x1 Start LBA 0xf9ffffff Size 0xd7a80001

or (with a 40 GiB boot volume and 2048 GiB autocarving — note the alignment shift accumulation):

LUN 0x0 Start LBA 0x0 Size 0x4ffffff
LUN 0x1 Start LBA 0x4ffffff Size 0xffffffff
LUN 0x2 Start LBA 0x104fffffe Size 0x583e0002

(Well, in this case the 0xffffffff autocarve size limitation is reasonable, but cannot be an excuse for breaking the alignment of the following volume — either the excess sector should be skipped, or the maximum allowed autocarve size should be set to 2047 GiB.)

The only option which works properly is tw_cli /cx add ... vol=a:b:c:d (with explicit specification of sizes for all required volumes except the last one). It gives properly aligned start sectors — e.g., for vol=2000:

LUN 0x0 Start LBA 0x0 Size 0xfa000000
LUN 0x1 Start LBA 0xfa000000 Size 0xd7a80000