1

for the second day in a row the system time on a VM I'm in charge of jumped forward by exactly 90026 seconds. In the afternoon or evening, not connectable to any cron maintenance job which typically run 3-5 am.

I don't know where to begin looking. /var/log/messages just says "systemd: Time has been changed", and ntpd of course quits because it's too scared to correct such a large deviation.

Any idea how to find out which process might be at fault?

musbur
  • 193
  • 12
  • You could use auditing to find the process that sets the time. You would first need to figure out which system calls to audit. – berndbausch Apr 30 '21 at 08:05
  • 2
    This likely means that your VM clock depends on host system clock. Check system time on your host (if you can) - if it is 90026 seconds away this is an issue. To know the exact solution you have to let us know what host type, what VM type and what virtualization type you use. – kab00m Apr 30 '21 at 09:14
  • Thanks for your suggestion. I can't access the host on which the VM is running and, frankly, I don't want to know anything about it. I've alerted my organization's IT department to this issue and they're looking into it. I just wanted to make sure that everything is OK on "my" side, which us the VM. – musbur Apr 30 '21 at 12:28
  • Upon reboot the VM's time is wrong by the same amount. Until the root cause is fixed I've set "maxchange 100000 1 -1" in /etc/chrony.conf to brute force the time if this happens again. – musbur Apr 30 '21 at 12:35
  • 1
    If you control the guest VM, you should be able to turn off time sync with the host. For VMware, there's an option in open-vm-tools; for Hyper-V there's a different service (can't remember its name right now). You'll still have to step the clock on boot, but at least it shouldn't jump whilst the machine is running. – Paul Gear May 01 '21 at 00:19

1 Answers1

2

Ninety thousand seconds is 25 hours, an enormous offset that sets the time to the wrong day.

The clock actually being stepped while the system is running, but chrony panics, suggests it was not done by NTP. Confirm your NTP servers are a correct reference by reviewing the output of chronyc tracking

Most likely remaining possibility is hypervisor syncing host time to the guest. Which happens at guest boot, resume, and possibly other occasions depending on platform. VM host time could be horribly wrong due to never having NTP set up correctly. Your compute administrators will need to point those hosts at NTP servers like every other device.

Yes, you can brute force correct it with chrony's maxchange. Beware stepping the time so far can make some applications misbehave.

John Mahowald
  • 32,050
  • 2
  • 19
  • 34