Centos Linux stuck in time

Question

Has anyone seen the Linux time being stuck, it has been driving me nuts since yesterday.

# date
Thu Apr  2 16:35:04 UTC 2020
# date
Thu Apr  2 16:35:04 UTC 2020
# date
Thu Apr  2 16:35:04 UTC 2020
# date
Thu Apr  2 16:35:04 UTC 2020

Here was about 3 minutes break while I logged into servfault.

# date
Thu Apr  2 16:35:05 UTC 2020

RTC seems to tick, this is about ten one-thousands apart:

# timedatectl
      Local time: Thu 2020-04-02 16:35:06 UTC
  Universal time: Thu 2020-04-02 16:35:06 UTC
        RTC time: Thu 2020-04-02 16:42:36
       Time zone: America/New_York (UTC, +0000)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a

# timedatectl
      Local time: Thu 2020-04-02 16:35:06 UTC
  Universal time: Thu 2020-04-02 16:35:06 UTC
        RTC time: Thu 2020-04-02 16:42:46
       Time zone: America/New_York (UTC, +0000)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a

# hwclock --show
Thu 02 Apr 2020 04:43:56 PM UTC  -0.001834 seconds
# hwclock --show
Thu 02 Apr 2020 04:44:04 PM UTC  -0.000916 seconds

These are also about ten secs apart:

# hwclock --hctosys
# date
Thu Apr  2 16:44:24 UTC 2020
# date
Thu Apr  2 16:44:24 UTC 2020

Try NTPd

# systemctl start ntpd
# date
Thu Apr  2 16:44:24 UTC 2020
# date
Thu Apr  2 16:44:24 UTC 2020
# date
Thu Apr  2 16:44:24 UTC 2020
# date
Thu Apr  2 16:44:24 UTC 2020

# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-04-02 16:44:24 UTC; 27ms ago
  Process: 18008 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 18009 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─18009 /usr/sbin/ntpd -u ntp:ntp -g



# ntpq
ntpq> lpeers
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 time.cloudflare .INIT.          16 u    -   64    0    0.000    0.000   0.977
 clock.sjc.he.ne .INIT.          16 u    -   64    0    0.000    0.000   0.977
 ntp4.doctor.com .INIT.          16 u    -   64    0    0.000    0.000   0.977
 tock.usshc.com  .INIT.          16 u    -   64    0    0.000    0.000   0.977

It just stays at .INIT. although peers are reacheable

# ping tock.usshc.com
PING tock.usshc.com (199.102.46.72) 56(84) bytes of data.
64 bytes from tock.usshc.com (199.102.46.72): icmp_seq=1 ttl=54 time=1.00 ms

Edit#1

# grep . /sys/devices/system/clocksource/clocksource*/[ac]*_clocksource
/sys/devices/system/clocksource/clocksource0/available_clocksource:tsc hpet acpi_pm refined-jiffies jiffies
/sys/devices/system/clocksource/clocksource0/current_clocksource:jiffies

# dmesg | grep clocksource
[    0.000000] Command line: vmlinuz nomodeset rw selinux=0 enforcing=0 raid=noautodetect nmi_watchdog=0 LAUNCH_INIT=/tmp/SUTup rdshell console=tty0 console=ttyS0,9600n1 HOMEBASE=192.168.1.1 biosdevname=0 clocksource=jiffies nmi_watchdog=0 consoleblank=0 vga=791 ramdisk_size=4000000 initrd=initramfs
[    0.000000] Kernel command line: vmlinuz nomodeset rw selinux=0 enforcing=0 raid=noautodetect nmi_watchdog=0 LAUNCH_INIT=/tmp/SUTup rdshell console=tty0 console=ttyS0,9600n1 HOMEBASE=192.168.1.1 biosdevname=0 clocksource=jiffies nmi_watchdog=0 consoleblank=0 vga=791 ramdisk_size=4000000 initrd=initramfs
[   51.409031] tsc: Refined TSC clocksource calibration: 2394.374 MHz

This server is getting really annoying. The running OS is a CentOS live image served via PXE. There's no FakeTime.

Check this out: this should take ~2 seconds but takes ten times that (running via SSH so I get an external 'time')

# time ssh 192.168.100.115 'for i in {0..1}; do date; sleep 1 ; done'
Thu Apr  2 23:39:22 UTC 2020
Thu Apr  2 23:39:23 UTC 2020
real    0m20.869s
user    0m0.026s
sys     0m0.007s

There's no firewall blocking NTP, it's a small network. It's a physical Intel server, not VM.

Edit#2

Paul suggested to check my clocksource, it was jiffies, switched to hpet, and things started working; not sure why it worked until yesterday (first time I noticed the time stuck), we've used this image for years on all sorts of hardware.

Clocksource: https://access.redhat.com/solutions/18627

Is a time altering shim like libfaketime installed? – John Mahowald Apr 02 '20 at 17:20 — John Mahowald, Apr 02 '20 at 17:20
Is your clock stuck in the future or in the past? – Jasen Apr 04 '20 at 08:42 — Jasen, Apr 04 '20 at 08:42

score 4 · Answer 1 · answered Apr 02 '20 at 23:37

4

Your peers might be reachable by ping, but they aren't reachable by NTP, which is the important thing for clock sync. You may have a local, network, or ISP-level firewall blocking access.

However, that doesn't explain your system clock freeze. Other than John Mahowald's suggestion, there could be hypervisor or firmware issues - is this a VM or some sort of exotic hardware?

What do the following commands show?

grep . /sys/devices/system/clocksource/clocksource*/[ac]*_clocksource
dmesg | grep clocksource

answered Apr 02 '20 at 23:37

Paul Gear

4,367
19
38

1

Paul, thanks for the suggestion, I changed from jiffies to hpet and seems to work now: `echo "hpet" > /sys/devices/system/clocksource/clocksource0/current_clocksource` – Christian Apr 03 '20 at 00:25
2

Glad to hear it's fixed. It looks like your kernel command line is set based on very outdated recommendations. I would remove the clocksource from the command line and let the kernel pick its preferred one (which would normally be `tsc` rather than `hpet`). – Paul Gear Apr 04 '20 at 02:50

Centos Linux stuck in time

1 Answers1