1

I have a 5 node Hyper-V cluster running.

All 5 servers are running Server 2012 R2, each node is attached two devices using iSCSI. The device I'm having problems with had a single volume that was used as a CSV. This was presented to the servers under c:\clusterstorage\volume2

I have gone through the process of moving all VM storage on to a new device, the new CSVs are up and running and nodes are connected. Everything for the new device is working correctly.

I then tried to remove the old device as a shared storage device by doing the following. I used PowerShell to get the paths of all VHDs on the cluster and none are pointing at c:\clusterstorage\volume2 I checked the configuration paths are not pointing at c:\clusterstorage\volume2 I also checked that the snapshot paths are not pointing at c:\clusterstorage\volume2

I started removing the iSCSI target on 4 of the nodes (The 4 the storage resources was not currently 'assigned' to) and this was successful. I then went to Failover Cluster Manager and set the CSV to offline. Instantly 2 of the VMs went into a saved state (luckily not critical servers). I brought the resource back online and started the machines. I inspected the settings page for each VM and there was no reference to c:\clusterstorage\volume2\ on either. I then went looking for the configuration files just to manually ensure something wasn't different between them and the settings page. I first looked in c:\clusterstorage\volume2\ the folder for either VM was not present here. I looked in the new locations c:\clusterstorage\volume5\ and c:\clusterstorage\volume6\ the folder for each VM was here. I checked the configuration files and they were exactly as they should be and didn't contain any reference to c:\clusterstorage\volume2\ I checked the C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines where I found the shortcuts for each VM pointing at the new volume locations.

In short nothing I can find points to c:\clusterstorage\volume2\ but if I take that CSV offline the two machines go into a saved state.

Even stranger I tried this again while writing this and yes the two machines went into a Saved State but I was able to start them again even though the CSV remained in a offline state. I now have two VMs where on the resource tab for each the storage shows as offline, but they are 'working'

So can anyone point me at a possible fix for this, perhaps a configuration file I've not found? Or would my best course of action be to delete the VMs while preserving VHDs and create a new VM and attach the original VHDs.

Sorry for the wall of text.

Drifter104
  • 3,773
  • 2
  • 25
  • 39

1 Answers1

2

I don't have much experience with Hyper-V in 2012 R2 but do have plenty with Hyper-V, Clusters etc. on 2008 R2 so will guess at the possible issue here.

I found that with Failover Clustering (in 2008 R2) when you add a VM into a failover cluster it will inspect the attached VHDs and the storage on which they reside and make a record of this in the cluster config so that they can be shown as dependent resources in Failover Cluster Manager (in the Storage pane when the VM is selected).

This does not necessarily reflect the current location of the VHD files - if they have been moved in Hyper-V since adding the VM to the failover cluster the dependency information is not updated. So you might have a VM that can access its disks just fine but failover clustering thinks it still has a dependency on the disk you are removing. When you offline the disk, it offlines the VM as a logical consequence. To solve (assuming this turns out to be the problem - have a look at the storage panel for the VM within FCM), remove the VM from the cluster and re-add it.

By the way as a best practice I would not generally start tearing down any iSCSI connections before you have removed the CSV from FCM.

Mintra
  • 561
  • 3
  • 7