In my organization, we are looking to virtualize servers (a dozen of Linux and Windows servers or so).
I am looking for a solution what would scale over the years and that would allow us maximum flexibility (i.e., moving virtual machines from one physical machine to another one when the old hardware becomes obsolete for example).
Which solution would you recommend in the short run and in the long run?
Citrix Xenserver Free is free. Note that this is a commercialised version of Xen, with management tools around it. It is not the same as installing your favourite distribution and adding Xen to it.
Citrix Xenserver Free gives you central multi-host management, no limit on physical hosts, resource pooling, SAN (iSCSI, FC, SAS) and NAS (NFS) shared storage, live migration between hosts (if using shared storage), P2V and V2V conversion, and so on. And it's free - no limits whatsoever. You can start with a couple of servers, and scale up and out as you grow.
Once you start needing more complicated infrastructure, you can upgrade to one of the pay-for solutions, which gets you high availability, VM provisioning, workload management, etc. It's not free, however :)
My experience with ESXi has been great. I'm currently using ESXi 4 on Core 2 Quad and Core i7 machines. The free license seems to support only 1 x 6-core physical CPU so if you have dual Xeon's you may want to investigate other free solutions.
Positive:
ESXi seems to run everything. I've run 32-bit and 64-bit versions of Linux, FreeBSD, Windows and even Mac OS X tiger successfully.
Memory overcommitment is a very pain reducing feature. Without this feature you have to micro-manage each VM's memory footprint or else the machine would not start. Most of the time the actual memory in use is far below the physical threshold and you can fire up more and more VM's. ESXi seems to scale to about 4 VM's per core as long as they are not all busy.
Scaling up to the enterprise is as easy as adding an ESXi host to a vCenter Server cluster. Enterprise features of vSphere are currently unmatched by competitors, e.g. vCenter can run two VM's in virtual lockstep allowing instantaneous failover without missing a cycle if a host dies. Another new feature is the automatic powering down of hosts during idle times. The running VM's on these hosts are moved off to other hosts.
Negative:
We did not have good experiences with VMware support. We ended up solving most problems ourselves. Maybe it's just because they're new in South Africa.
You have to be careful with cheap hardware. It's certainly possible to get it to work on hardware not on the HCL list, but you'll have to do some homework. The most important to look out for is NIC, SATA and 64-bit guest support. For 64-bit guest support your host must have hardware virtualization. Note that installing ESXi on the hard drive does not guarantee that you're going to be able to create VM's on it.
Windows Vista and Server 2008 caches very aggressively. This does not work well with overcommitment of memory because Vista will use all memory assigned increasing memory pressure. This does not affect Microsoft's Hyper-V since Hyper-V does not offer overcommitment. Vista and Server 2008's behaviour will probably change in the next version of Hyper-V since overcommitment is on the road map for Hyper-V.
Microsoft support will probably require reproducing the error on physical hardware before they will look at it. The good thing is that those Microsoft products that are supported on Hyper-V are usually also supported on ESX.
Given the small number of servers you are considering managing I would go with the VMware vSphere Essentials bundle. At only $995 it would give you the ability to scale to 3 physical ESX/ESXi servers (which could easily support dozens of VMs depending on the HW you select) and provide you with a best-in-class central management solution (VirtualCenter now called vCenter) to manage all of your ESX hosts and virtual machines from a single console.
Should you decide you require high-availability and an integrated backup solution for your virtual machines you can upgrade to the Essentials Plus bundle with no re-installation (you just enter a new license key). It even gives you a simple migration path to VMware's Enteprise feature sets should your needs change in a few years.
Here's a link on VMware's site with more information about the SMB targeted bundles: www.vmware.com/vmwarestore/vsphere_smbpurchaseoptions.html
I prefer Xen over VMware, as it's less of a black box and is, as a result, far more flexible. That does result in it having a reputation for being harder to manage, but given the fun I'm seeing with trying to "manage" some ESX servers at work, I'm not sure that's all that justified.
The other nice thing about Xen is that the way you run VMs is quite similar to how you run VMs under KVM, so as that project matures, if you decide to move there the migration will be trivial.
Having run numerous virtualization farms, both for big business and personal use, I'm of one singular camp. VMware. The VMware tools support is currently unparalleled, between Lab Manager, LifeCycle manager, etc. Citrix XenServer is coming up, but it has 10 years of makeup work to do to catch up with the mature VMware products.
And as a general rule I've found that even though you start at a dozen or so, you need to plan for 50. Once you start virtualizing, you never stop.
At the moment, though the Citrix offerings are promising, VMware is your best choice for the near future. If you're willing to live with the growing pains Citrix offers (immature tools support), I don't think you could lose with either open.
And both have Free-as-in-beer versions you can try-before-you-buy.
The one downside of Xen that we picked up on a few months ago was that Xen cannot overcommit memory, so you end up with less memory if you're trying to achieve high density.
VMWare can single instance some memory objects, which allows VMWare to overcommit memory to virtual machines. On the other hand, VMWare gives me the same annoying feeling the EMC and NetApp do -- all of those cool features require yet another license.
Clarification: You end up with less memory allocated per VM. We were looking at Xen in the context of desktop virtualization. The Citrix model is to allocate very small (256MB) Windows XP VMs, then use XenApp (aka Citrix Presentation Server) to actually deliver applications. VMWare will lie to the desktop and tell it that 1GB is available, and run applications in the VMs compute space
I currently run ESXi 3.5 U2. I've been hesitant to go to 4.0 because the licensing appears to hint at a timeout for the free version, but isnt clear. Plus when I installed version 3.5 U4 I got a purple screen due to a CPU versioning issue in my Opterons. I run 4 quadcore opterons on ESXi FREE. (16cores x 64bits). With 16gb of physical memory I am running 24 VMs on the one box. This is a self contained box with 20 SataII Hds. The Largest array is a RAID60 of 1.8TB. The servers are running great. Mostly Ubuntu, 5 Solaris, and one Windows. I use the "unsupported" SSH and SCP and VMCLI from Windows with perl scripting to run backups - very slow. Working on a datastore to datastore copy process with hotswapping the drive for backup or secondary streaming via SCP as a faster process..
I tried Xen, and now wish I have given it a better chance as my Adaptec ICP needs on board Java powered application to manage the array. ESXi does not support this, Xen does.
So, I must rely on the autofailover features of the controller and the hot spares for recovery. Two days ago, the server took a power surge. UPS fried. The surge broke the array. I had to use BIOs tools to recover... Somehow this was successful...
Currently running a new set of backups for the VMs.
Still considering Hyper-V. But Xen has moved up on my list and I plan to build a test server soon to try to migrate some VMs to.
Being that none of my projects are bringing anything in that comes close to paying the internet bill or datacenter fees, I would be hard pressed to put the kind of money VMWare wants out there..
But in answer to the question above about number of procs/cores. Yes, 4 procs, 4 cores each. no problem. But this is not vSphere 4 either and there did appear to be some licensign changes.
I have 2 x 32 cores with 128 Go RAM on it and running on Citrix Xen. We enxperienced too muchs stability problems, licensing problems with version ugrade, no good support and multipath is working bad.
I'd like to go through another solution like vmware, but i wonder if virtualisation is enough mature for Production.
@Hans (since I can't comment yet) - When you get your license key from VMWare you can specify how many physical CPU sockets to get it for. If you only said 1 when you obtained the key that may be why it doesn't see your other CPU. I believe the limit is 2 sockets though. I also have a dual quad-core box with ESXi 3.5 and that is what I had to do.
The question may be decided for you depending on what Linux distributions you are planning to run. If they are supported by XenServer I'd recommend that. You can start with the free version, which supports various shared storage types, live migration, snapshots, backup & restore, and work you way up as needed. It would probably cost less in the long run since VMware's prices are astronomical once you grow past the essentials. Want cool features in VMware? You're gonna pay for them! ;)
That being said, if your Linux distributions are not supported by XenServer I'd recommend VMware...non-Xen guests running in Xen perform terribly. VMware has been around the longest and is the most mature. As such it has the largest list of supported guest OSs.
If you're a Windows shop you might want to look at Hyper-V as well. Though it has a small list of supported Linux guests, they seem to run faster than in XenServer. If you run Hyper-V Server on a domain management is fairly easy, but I wouldn't recommend it if you aren't on a domain. It should run on just about any hardware with Windows drivers, and the upcoming R2 supports clustering and live migration even down to the free Hyper-V Server!
To be honest, both of these are very viable platforms over both short and long term, currently the two best technology leaders and the like.
We've gone with VMWare and we're very happy with it but at the end of the day it's subjective imho.
