NTP server architecture

Question

I have a virtual environment running several Linux boxes and I'm planning how to manage all of the ntp architecture.

As far as I know, there is no use in having two servers in 'ntp.conf' file, there should be only one or more than three ntp servers pointed by the client so, my first approach is having one server 'server1' pointing to 4 public servers, specifically RHEL ones, then having other box 'server2' pointing to server1 and below all my other Linux servers pointing to server2.

But I have observed weird behaviour with this architecture. I've seen some servers desyncing between server2 and them and even sometimes server1 and server2 aren't perfectly synced.

My first question is, why is that happening?

Then I came up with another architecture which was having the same server1 pointing to public ntp servers and then having three servers, 'server2', 'server3' and 'server4' pointing to server1 and below all my other machines pointing to servers2-4.

• Is there a chance this architecture could improve the syncing within all my network?

• Or would it be the same performance between syncs? What would be the best way to architecture this?
  

Edited

Here is the output of ntpq -p from server1:

remote          refid      st t when poll reach   delay   offset  jitter
=========================================================================
*Time100.Stupi. .PPS.       1 u  317 1024  377  182.786    5.327   3.022
LOCAL(0)        .LOCL.     10 l  46h   64    0    0.000    0.000   0.000

And here its ntp.conf:

# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.rhel.pool.ntp.org iburst
server 1.rhel.pool.ntp.org iburst
server 2.rhel.pool.ntp.org iburst
server 3.rhel.pool.ntp.org iburst
#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

# Enable writing of statistics records.
statistics clockstats cryptostats loopstats peerstats sysstats rawstats

### Added by IPA Installer ###
server 127.127.1.0
fudge 127.127.1.0 stratum 10

Here is the output of three of the clients:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*server1         172.16.29.21     3 u    1   64    1    1.090   -0.138   0.036


     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*server1         172.16.29.21     3 u 1035 1024  377    1.117   -1.943   0.530


     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*server1         172.16.29.21     3 u   32   64    1    0.902    1.788   0.140

Answer 1

Depending on how critical time keeping is in your environment, you may not want server1 to be a single point of failure. If you have to take it offline for maintenance or repair for an extended period of time, its peers will stop syncing. It is all downhill from there.

Why not have server1, server2, server3, server4 all sync to 4 or 5 Internet peers. Then, your internal network can reference these systems?

Conventional wisdom says that 3 is what you need for quorum, but you need to be tolerant of at least one being determined as a falseticker or going offline.

Please see; 5.3.3. Upstream Time Server Quantity

Additionally, you mention weirdness and issues with your current configuration. It would help to see the output of ntpq -p for the relevant hosts.

Answer 2

While it's not strictly true that 2 servers is no use, the NTP Best Current Practices RFC (8633) recommends 4 as a minimum. NTP's intersection algorithm doesn't depend merely on quorum in the number of servers, but also in the quality of the time which they return - and you can't predict that. So the more the better. There is no problem having up to 10 upstream NTP servers.

As Aaron mentioned, your proposed servers 1-4 should all point to upstream NTP servers, and your internal systems should point to all 4 of them. Servers 1-4 can also peer with each other (in symmetric mode), but that's not strictly required.

It's important to understand why you shouldn't funnel NTP through a single server at any point in your architecture: NTP requires multiple servers for accuracy, not just redundancy (see the NTP docs for a description of the algorithms, which explains why). (Shameless plug: I've written more about this elsewhere, including suggestions for architecture.)