What does your SMB VMware cluster look like?

Question

We are a SMB looking to bring on a second VMWare server. Currently our primary server is running local VMDK images. Assuming we have two nicely sized machines (dual quad xeons, 16gb ram), and abuot 6-10 instances to run supporting 50 users -- what sort of SAN should I look at?

EMC Clariion/Equalogic was ~30k for 4tb raw. Is there a decent benchmark that might help me decide if we need a 30k 2tb (raided) sana, or if there are lower end systems that are more value minded for a SMB?

It looks like vReplicator would allow for VMDK replication -- not true HA or vMotion, but with local storage.

Answer 1

If you want to stick to the EMC platform you should look into the AX line. Dell sells them all the time. They are a slimed down verion of the CX line running on all SATA disks (might be able to get SAS drives for them). Much cheaper than a full on CX server and I think they support FC and iSCSI).

As to what ours looks like (we qualify as an SMB) we've got an EMC CX4-240 with 105 spindles and two Dell 905s with 64 Gigs each running about 60 VMs on them (plus a few physical servers that weren't good candidates for VMs).

Answer 2

iSCSI can be a viable alternative to a "traditional" fibre channel-based SAN.

I have a Customer using an iSCSI SAN between two VMware cluster nodes. I wasn't involved in spec'ing the solution, so I can't comment on what considerations went into deciding what to purchase (though I don't think that they actually did much homework). They're using a Dell NX1950 running Windows Server 2003 Unified Storage Management Edition with a DASD Dell MD3000 cabinet.

If I were spec'ing such a thing I'd do some benchmarking of your I/O throughput needs between the VMs and the disks. There's a nice article here on optimizing VMware 3.X and iSCSI that can give you some details about considerations re: iSCSI and throughput.

For my SMB Customer, if I were to be involved in the initial spec'ing, I'd have looked at what the EqualLogic products would've offered in lieu of the NX1950 and the overhead of and expense of running Windows on it. Perhaps it would've been a wash-- I don't know.

Answer 3

Are you looking to go iSCSI or Fibre? Nice solution based on Solaris Kernel and ZFS filesystem:NexentaStor

Dell MD3000i and IBM DS3300 have nice iSCSI solutions, but I found them limiting on snapshots and licensing for those. (Both based off the LSI Engenio array)

Ensure you get plenty of cache and dual controllers for redundancy. For the NexentaStore, get lots of smaller/fast drives for some serious IOs.

Answer 4

Equallogic's architecture is pretty sweet in terms of ease of use and performance (with multiple arrays at any rate) but it chews up raw capacity fairly aggressively. If you want to make full use of hardware snapshots for example just following Equallogic's recommended reserve numbers 4TB of raw capacity becomes about 1.8GB (RAID 5) or 750G (RAID 10) and my experience of them is that you ignore those recommendations at your peril.

Cost wise the PS6000's aren't exactly cheap but the new "entry level" PS4000's seem to be a nice trade off (much lower cost, half the bandwidth). The $30k you mention was almost certainly for one of the PS6000 range, I'd go back and ask about the PS4000 as it should be a lot less than any Clariion solution with a similar capacity.

One major issue with iSCSI and VMware ESX 3.5 (and earlier) is that the VMware iSCSI software initiator (if you are using it) is pretty weak - doesn't support TOE\iSCSI offload, is a single threaded stack with meagre multi-path capability so there's no way to get Native ESX iSCSI to utilise more than 2Gbps of bandwitdh. iSCSI HBA's can help a bit but the best solution is to move to ESX 4 ASAP - the new iSCSI initiator is substantially better and the revised pluggable storage architecture supports proper dynamic multipathing for iSCSI. This will mean substantial performance and scaleout improvements for Equallogic (for those environments that need lots of bandwith\IOPS).

Expanding these in the future is extremely simple too and their AutoSnapshot Manager / VMware Edition can do some pretty neat stuff in terms of replication although you will need multiple arrays to get the most out of it. One important thing to remember about Equallogic is that all of the software features (including the MPIO drivers and ASM) are "free" - this is not true for EMC kit and those costs add up very fast.

One other solution to consider might be something like LeftHand's Virtual SAN Appliance. I think it costs around $3K for up to 2TB of shared storage - it basically allows you to use the local storage on your ESX hosts as a virtual SAN. I can't comment on performance or scalability but the concept is pretty neat.