4

In the last two months, we have got quite a few issues with the server time and date occasionally changing to some absolute random values. Sometimes it is a few days in the future or past, sometimes - even several months.

We have got the problem on several VMs, so it is not the problem with one VM unfortunately.

We run Windows Server 2019 (Version 1809 OS Build 17763.3406) on Amazon EC2. We synced time using AWS default endpoint 169.254.169.123, and then we switched to managed AWS Active Directory. The provisioning is done with Ansible, not sure if it matters though.

All time change events are logged in the system, and look identical, like this one:

The system time has changed to ‎2022‎-‎12‎-‎23T01:10:04.150070300Z from ‎>2022‎-‎09‎-‎14T12:22:57.146781800Z.

Change Reason: An application or system component changed the time. Process: '\Device\HarddiskVolume1\Windows\System32\svchost.exe' (PID 372).

As you can see, the change is very significant and thus, affects the web applications run on IIS server on these VMs.

Do you know what can be the reason, and where we should look in order to fix the issue?

Edit: adding details according to @Greg's comment:

C:\Windows\system32>w32tm /query /status /verbose Leap Indicator: 0(no warning) Stratum: 6 (secondary reference - syncd by (S)NTP) Precision: -23 (119.209ns per tick) Root Delay: 0.0643878s Root Dispersion: 7.8759466s ReferenceId: 0x0A050C5D (source IP: 10.3.15.74) (edited) Last Successful Sync Time: 11/2/2022 11:05:39 PM Source: WIN-BOLJH2K6OOL.companydomain.cloud (edited) Poll Interval: 6 (64s)

Phase Offset: -0.0023147s ClockRate: 0.0156250s State Machine: 1 (Hold) Time Source Flags: 8 (SignatureAuthenticated ) Server Role: 0 (None) Last Sync Error: 0 (The command completed successfully.) Time since Last Good Sync Time: 8.5343114s

MyUserName
  • 91
  • 5
  • 1
    You will probably need the output of `w32tm /query /status /verbose` – Greg Askew Nov 01 '22 at 16:13
  • 1
    If I'm reading that correctly, the erroneous time came from `WIN-BOLJH2K6OOL.companydomain.cloud (edited)` at address `(source IP: 10.3.15.74) (edited)`. Maybe you can tell the time service to not look at that machine? – tsc_chazz Nov 17 '22 at 22:27
  • If you look at the full command line of the process reported with the specified PID, what does it say? Does the PID match any running services? – Ferrybig Nov 22 '22 at 13:40

2 Answers2

1

Try using an external time server. I recommend: pool.ntp.org

See: https://www.pool.ntp.org/de/use.html

We use pool.ntp.org as time source of our Domain Controllers and all devices which are not in the AD (linux machines, switches, printers, firewall, ...) All windows machines use the DCs as time source. That is very important!

If changing the time source is technically not feasible for you, maybe this helps you (enhanced logging of time-related events in newer Windows versions since Mid 2021):

https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/windows-time-for-traceability?tabs=257

This should help to find out what happens and why.

Glorfindel
  • 1,213
  • 4
  • 15
  • 22
RebootDeluxe
  • 121
  • 6
1

There are a couple things that will cause this that I have experienced. One is the bios time is set incorrectly. Second is the CMOS battery dying or on the way out.

mAineAc
  • 11
  • 2
  • 1
    Your answer is not helpful in this case. This is because @MyUserName is using VMs on Amazon EC2. Therefore he has no control over CMOS batteries and bios time. – RebootDeluxe Nov 23 '22 at 10:27
  • Yeah, it's not about some local machines, they are VMs in AWS, and there are several of them, so I don't think the issues is in hardware. – MyUserName Nov 30 '22 at 08:15
  • Ahh I misunderstood. I thought this was a local server – mAineAc Dec 09 '22 at 02:08