2

One of our production servers was "cloned" to make a secondary test environment. The source server is a virtual machine (VMWare vSphere environment) joined to a domain and with a static IP address.

When the cloned VM was powered on, the virtual network adapter was enabled. The computer name was not changed. Neither was the IP address. Somehow, those two servers with the same DNS name and IP address co-existed on the same domain for several days--maybe as much as a week. We'll be busy undoing the damage for a while...

Whether it's carelessness or ignorance, people make mistakes. I can accept this. So let's assume it might happen again. I know how to fix the problem. But how do I detect it?

EDIT
I am not looking for instructions or advice on how to clone a VM. I am not the one given that task. I want to know how to quickly (or immediately) detect when two computers/workstations/VM's have the same DNS name.

UPDATE
I decided to run a "check" at startup of the main SQL Server service that would try to determine if the SQL host had been cloned. (Note that this tactic only applies to VM's that are SQL Server hosts.)

TLDR: Hard-code the SQL host machine name and domain in a stored proc and compare those values to SERVERPROPERTY(N'MachineName') and DEFAULT_DOMAIN(). Do something drastic if the values don't match.

If anyone is interested, I blogged about it: SQL Server: Attack Of The Clones

Final Thoughts
I probably should have mentioned this when I originally posted the question: I am a SQL Server DBA. Even though I'm not a sysadmin-type, I do work with them. I'm not on the same "team" as the sysadmins, but I'm not walled off from them either. I appreciate all the input and numerous answers to my question. Most have indicated the problem is a training issue and I should educate those involved. I can't disagree. This is a sensible, proactive approach. But...almost everything outside of the SQL Server realm is beyond my control. Sysadmins come and go. They act and make decisions independent of my desires and needs. However, I do have control over what happens from the SQL Server perspective. I can still be proactive, even after the deed is done. Since my home-grown solution is SQL Server-specific, now it's apparent I should have posted my question on https://dba.stackexchange.com/ I think there's still some value to having the question here, but if a moderator wants to migrate it, I understand.

Community
  • 1
Dave Mason
  • 145
  • 1
  • 5
  • 1
    Well, for one thing, one or both machines would be complaining about an IP address conflict. You should *always* be using sysprep when cloning Windows VMs. – Nathan C Nov 20 '14 at 02:47
  • 3
    @NathanC - I don't this was a "clone" in the sense of a VM cloning operation. I suspect this was "let's make a copy of a production VM and run it in an isolated environment to do some testing". The isolated environment wasn't. – Evan Anderson Nov 20 '14 at 02:48
  • `We'll be busy undoing the damage for a while` - what damage? – joeqwerty Nov 20 '14 at 03:05
  • 1
    @joeqwerty: the cloned server was a SQL Server host. I had two servers running identical backup jobs. I now have no idea if any of the db backups from the last week are good or not. Periodic backup jobs for exactly one of the two servers would fail every time. The email alert I received referenced the DNS name of the server, so I don't know which server "won". I have to compare data from both servers and try to determine if sql transactions were applied to both servers or one server. If it was one server, was it consistently the same server? Etc. – Dave Mason Nov 20 '14 at 03:08

4 Answers4

5

When you clone a virtual machine in a vSphere environment, you have an option to "customize guest" when cloning or deploying VMs from a template...

This is where you can specify a name, change network settings, specify domain membership and generate a new SID (for Microsoft operating systems). That's all you need to do going forward.

ewwhite
  • 197,159
  • 92
  • 443
  • 809
  • That's good to know. I'll pass that along to our system admins. But I am not the one who clones environments. I wanna know how to immediately detect a "bad" clone when someone else creates it. – Dave Mason Nov 20 '14 at 03:11
  • 7
    It shouldn't happen. You shouldn't be having "bad clones". It's irresponsible and sloppy to create VM clones without *at least* changing the IP address or computer name. The SID can be the same, unless you're cloning domain controllers... but then, you shouldn't be cloning domain controllers :) - Try to focus on correcting the process and the root cause instead of fixing the symptom. – ewwhite Nov 20 '14 at 03:13
4

There most likely were logs in event viewer that indicated IP conflicts. Any monitoring system that gathers and filters event logs should be able to detect that and trigger alerts.

On physical managed switches you could probably detect and warn about the duplicate MAC address - I am not familiar enough with vmware virtual switches to know what options you might have there.

Of course the real solution is proper training for everyone involved in cloning to make sure it doesnt happen in the first place - cloning a VM needs to be done very carefully, changing the IP or moving to an isolated network to avoid this exact situation.

Hopefully one machine "won" for the whole time and changes werent spread across both servers!

Grant
  • 17,859
  • 14
  • 72
  • 103
  • I spent some time looking at the event log on the "source" VM. There is a monitoring system in place which runs as a Windows service. I found dozens of entries indicating that the service failed to start. Interesting, eh? I never saw anything about IP conflicts, though. I can conduct the same search on the clone. Even if I found something, I'm not sure how that would help going forward. If the situation repeated, I wouldn't know to investigate the clone until after the problem was discovered. Catch-22, it seems. – Dave Mason Nov 20 '14 at 03:53
  • 3
    @DMason Yeah, the only way to prevent the situation before it happens is by preventing the situation before it happens. Once there is an IP conflict, the problem has already happened. So like ewwhite said, you need to correct the broken process that lets this happen in the first place, rather than trying to clean up the symptoms after it's already happened. – HopelessN00b Nov 20 '14 at 03:57
2

There is no substitute for good IP management policies, ever. The End. Fin. FIN-ACK.

Regardless of context, there is no way to prevent people from being stupid with IP addresses unless the software goes out of its way to forbid it -- and it generally can't. How does it know you don't need multiple VMs to hold those addresses, and will only ever online one of them at a time? It's not the software's job to police this. It's the admin's job to value their job enough to not create an outage by creating an IP conflict.

This is one of those lessons that admins have to learn, either the hard way or the inevitable 3AM panics that happen when other people are learning the hard way.

  • You do not screw around with risks for IP conflicts.
  • The care you place in working with something holding the IP address of something important should be greater than or equal your desire to continue receiving paychecks.
  • More importantly, you cannot assume that something will prevent or detect them for you -- other than your own diligence and genuflection before your organization's anointed source of IP Truth.

tl;dr, no one is ever smarter than the IPAM. Assigning an IP address or cloning a device is a religious function that must be accompanied by the IPAM's blessing. Belief in heathenly practices designed to prevent or mitigate IP conflicts will only result in ordained ruin. Ignore its divine guidance at your own peril.

(I get what you're trying to do. I really do, and I respect it. But there is no true solution other than fear on the part of those who wish to keep their jobs.)

Andrew B
  • 32,588
  • 12
  • 93
  • 131
1

I agree with the comments that the tech creating these clones needs some re-education. Maybe some push back on your part, especially if the test servers are on the same network as production servers.

In the mean time, this might help you to identify VM guests with duplicate IPs already in your environment. You will need PowerCli, and keep in mind that cluster members use the same IP by design.

$vcserver = "your-vcenter-server-name"
Add-PSsnapin VMware.VimAutomation.Core
Connect-VIServer $vcserver
$results = Get-VM | Select -ExpandProperty Guest | % {
   $name = $_.HostName
   $_.IPAddress | % {
      New-Object PSObject -Property @{
         HostName = $name
         IP = $_
      }
   }
}
Disconnect-VIServer

Then find duplicates with this:

$count = @{}
$results | % { $count["$($_.IP)"] += 1 }
$count.Keys | ? { $count["$_"] -gt 1 } | % {
   $dup = $_
   $results | ? { $_.IP -eq $dup }
} | ft
xXhRQ8sD2L7Z
  • 695
  • 5
  • 12