How to deal with clock skew in a multi-node Kubernetes cluster?

Question

The issue is that the logs for a specific request (identified by a traceId printed in each log line) are being collected by loki-stack in a Kubernetes cluster, but the logs are not being displayed in the correct order. This is because the microservices that generate the logs may be running on different nodes, and the clocks on these nodes may have a skew, resulting in inaccurate timestamps. As a result, the logs appear out of order.

While it is possible to minimize clock skew through techniques such as NTP time synchronization, it is difficult to guarantee that the skew will be low enough to ensure proper log order, especially for requests that are processed in milliseconds.

Answer 1

The original NTP daemon, ntpd, has been joined by a newer one, chronyd. Both keep the local host's time synchronized with the time server. Both services are available, and I have seen nothing to indicate that this will change anytime soon.

Chrony has features that make it the better choice for most environments for the following reasons:

• Chrony can synchronize to the time server much faster than NTP. This is good for laptops or desktops that don't run constantly.
• It can compensate for fluctuating clock frequencies, such as when a host hibernates or enters sleep mode, or when the clock speed varies due to frequency stepping that slows clock speeds when loads are low.
• It handles intermittent network connections and bandwidth saturation.
• It adjusts for network delays and latency.
• After the initial time sync, Chrony never steps the clock. This ensures stable and consistent time intervals for system services and applications.
• Chrony can work even without a network connection. In this case, the local host or server can be updated manually.

Source: opensource.com