Dell T710 time issue, clock loses time every day

Question

I have a Dell T710 here at work that for some reason will not keep proper time. Its a dual quadcore with 32GB ram and runs Windows 2008 R2. Instead of Having multiple servers we run a few virtual machines under Hyper-V. All of the VM's have "time synchronization" enabled and the VM's clocks keep perfect pace with the host clock.

One of the VM's hosts our shop control software and since it utilizes the system time for time stamping actions, its vital the clock be as accurate as possible. Every day I have issues with the clock being slow. In the morning it is usually 2 or 3 minutes slow or on some days (like today), it was a full 10 minutes slow. If I set the server time using the internet time application, the time sets itself correctly and then the workstations follow through with NTP. It works fine but even toward the end of the day the clock is off yet again by 1 or 2 minutes.

The problem is maddening and I have tried everything I could think of and find on the internet including:

• Trying multiple internet time servers
• Using one of those atomic clock programs to adjust the frequency of updates to twice per day
• Manually tweaking the registry keys
• Ensuring that the VM's derive their time from the host which means that the clock drift is related to the host and not the VM's as every VM has the same time the host displays. (e.g. if the host clock is 7:28, all VM's show 7:28)
• One of the VM's is a domain controller BUT the host is not on the domain. This was done to ensure the host is just that, a host for the rest of the network and not part of it.

NOTHING stops the clock from drifting. From my research it could be linked to the fact that the the server load might have an effect on the time but the load is nothing at all, maybe spikes of 10-50%, its a small company. And for the reference time I simply look at my cell phone.

Answer 1

1. Disable host time synchronization.
2. Point VMs at an accurate NTP server.
3. Profit.

If host synchronization is enabled, all the other changes you do aren't going to make a difference, as the guests are going to try to keep syncing up with the hardware time on the host, which is evidently not keeping good time. Effectively, the host-time-synchronization over rides whatever time changes you're making, so turn it off and save yourself a bunch of headache.

For whatever it's worth, I always disable guest-host time sync (with any hypervisor) and point all my servers at a proper NTP source. In a domain, that's done by pointing one DC at an NTP pool, and the other domain machines at it. If not domain-joined, you may as well punch up a script to point all your machines at the same NTP server or pool.

Answer 2

Since you stated that the VM host is not a part of the domain this sounds like you may have a hardware issue, specifically you may want to make sure that your motherboard on that server is storing and counting time accurately.

As a starting point I would schedule a script to run periodically for a day to see if you can determine if there are other events taking place that may impact it. As a sample you could schedule the following command to dump to a text file every hour for 24 hours, it will record the offset from your NTP sources.

C:\> w32tm /monitor

You might want to post a handful of the results back here after that but you may also want to talk with Dell to see if there are any known issues regarding the CMOS or time drift with the hardware, they may have a BIOS fix or send a tech out to replace hardware.

Lastly, you don't have to segregate your VM hosts to be off the domain, as a matter of fact you may want it on the domain in the future if you ever have a need to get into clustering and High Availability being a member of the domain is required. You will need to make sure you have a physical DC as part of that effort though.

Answer 3

SOLVED

Interestingly enough, a simple fix solved my problem. The server runs 8 VM's, none of them are high load (infact mostly idle or test VM's) and two are the most important: the Domain Controller and a 2003 Terminal Server for shop control software. I decided to change one VM each day until I saw any change in behaviour. I started by disabling "Time Synchronization" for the 2003 server and since then I have not had any clock drift. Its been 3 days and the clock is 3 seconds faster than my cell phone, not bad at all. Looks like my first try was a lucky guess.

But I still can't figure out why the host clock would drift because of the VM. I don't believe there was actual drift but rather the TS was somehow entering a kind of infinite loop of trying to set the time from the host and somehow that screwed up the host clock. Or the TS was somehow setting the host clock (isn't time syncing one way?) causing a slight drift and the DC picked up that drift and it propagated through NTP back to the TS. I can't explain it.