1

I have a private server on AWS that has no publicly open inbound ports. The only open port is SSH, and it's only accessible from other machines in the same subnet.

This machine reads files from S3 over the network. Recently it stopped working because the server's time had drifted by more than 15 minutes and the requests to S3 were rejected:

<Code>RequestTimeTooSkewed</Code>
<Message>
    The difference between the request time and the current time
    is too large.
</Message>
<MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds>

The easy fix is to install ntp, but it requires opening UDP port 123. Because UDP is stateless, both inbound and outbound ports must be open.

How can I automatically update the system time using only TCP connections initiated by my server? Is there a standard TCP-based daemon like NTP? I don't care if it's not as precise: even a skew of up to 10 minutes would be acceptable.

Edit 2 Jun 2017

This is how the security groups are set up:

  • Inbound

    Type           Protocol    Port Range    Source
SSH            TCP         22            172.31.0.0/16

  • Outbound

    Type           Protocol    Port range    Destination
All traffic    All         All           0.0.0.0/0

This is the network ACL for the subnet - just the default ACL:

  • Inbound

    Rule #    Type           Protocol    Port Range    Source       Allow / Deny
100       ALL Traffic    ALL         ALL           0.0.0.0/0    ALLOW
*         ALL Traffic    ALL         ALL           0.0.0.0/0    DENY

  • Outbound

    Rule #    Type           Protocol    Port Range    Destination  Allow / Deny
100       ALL Traffic    ALL         ALL           0.0.0.0/0    ALLOW
*         ALL Traffic    ALL         ALL           0.0.0.0/0    DENY

Edit 2 Jun 2017 #2

Alright now it works without any special security groups, as prediceted by @Tim. I think I was just not testing it properly:

  • ntpdate doesn't use the servers in /etc/ntp.conf, so it was reporting an error:

    no servers can be used, exiting

  • ntpd does not attempt to update the clock as soon as it starts; it waits for a minute or so.

Testing instead with ntpdate-debian, and without port 123 open in a security group, works fine; and ntpd updates the time properly if I let it run for a while.

z0r
  • 165
  • 2
  • 9
  • Put a note in your calendar, once a month / quarter / year log in and manually set the time. – Tim Jun 02 '17 at 00:56
  • @Tim that doesn't sound very automatic ;) – z0r Jun 02 '17 at 00:57
  • I know it's not automatic, but it's practical, if a little time consuming. 10 minutes drift probably takes months. Would you consider opening UDP to only a specific IP address? – Tim Jun 02 '17 at 01:23
  • Not sure why you worry about the rules. It is probably best to have your own ntp proxy, but anyway if you want to use tcp you could use tlstime: https://linux-audit.com/tlsdate-the-secure-alternative-for-ntpd-ntpdate-and-rdate/ – eckes Jun 02 '17 at 06:23

2 Answers2

11

According to the AWS Documentation you can open UDP:123 in your security group outbound only. Because security groups are stateful replies will get back to you, but no-one outside your VPC will be able to initiate a connection.

You will of course require NACLs open in both direction for that port.

Update You should read about AWS Security, particularly security groups and NACLs.

NACL is network ACL. This is a firewall that lives outside your instance and only allows traffic to reach your instance if you have ports open. By default all inbound and outbound ports are open, but you can configure any way you like. NACLs are stateless so you need to open ports in each direction. Don't forget ephemeral ports.

Security groups on the other hand enforce network rules on the hypervisor level. They're stateful so if you allow an outgoing port then the response is automatically allowed back into the instance.

NACLs in theory reduce host server load slightly, because they prevent traffic hitting the network interface. Since you only control a small fraction of the server that probably doesn't make much difference.

Update 1st December 2017

AWS have announced the AWS Time Sync Service. In short, they're providing servers inside each data center to ensure the server time is accurate.

The AWS documentation says to use the Chrony software, rather than NTP, but both the NTP service. There's no point me copying the AWS instructions into this post as they may change in future, and AWS documentation is excellent.

Using NTPD

Instead of installing Chrony I've simply added the following to my /etc/ntp.conf to try to get NTP to use the new AWS NTP server

server 169.254.169.123 prefer iburst

I also made sure none of the other server statements had "prefer" configured.

Tim
  • 31,888
  • 7
  • 52
  • 78
  • Oh that's interesting! I didn't know about NACLs. Can you explain how I would open that port in the ACL? Otherwise I'll look it up. – z0r Jun 02 '17 at 01:32
  • Answer updated with background. If you don't know about NACLs they're all open, and you're likely using security groups to control access to your instance. Please edit your question to add information about how you're currently controlling access, tag me here, and I'll try to help more. – Tim Jun 02 '17 at 01:38
  • I've updated the question. But it's weird; now it seems to work even without the UDP port open in the security group, as you predicted. Maybe I was confused because I was testing with `ntpdate`, which apparently [doesn't use the config file](https://askubuntu.com/a/148207/81211). Oops :( – z0r Jun 02 '17 at 06:30
  • Your security groups and NACLs are allowing unrestricted outbound traffic. I suggest you do some more reading on AWS security because your statements don't match your configuration. – Tim Jun 02 '17 at 09:27
  • 1
    [Chrony](https://chrony.tuxfamily.org/) is another implementation of NTP, just as ntpd is another implementation of NTP... – Michael Hampton Nov 30 '17 at 21:03
0

If you're not opposed to running a cron job on another machine that has accurate time then you can always have one automatically update the date over SSH. If you have SSH permissions on another server that has an accurate date you can do it the other way as well.

user418378
  • 29
  • 3
  • That's a nice idea, but it adds significant complexity managing keys. I would probably implement my own HTTP-based time server instead. Also, it still requires that other machine to keep accurate time, probably with NTP. – z0r Jun 02 '17 at 01:16
  • Ha, S3 actually returns the current time in its error response. I could use that, if I wrote a cron script to do so. I was hoping for something off-the-shelf tho. – z0r Jun 02 '17 at 01:17
  • If UDP is blocked that rules out NTP, and I'm not sure if there are any off the shelf solutions besides NTP for time synchronization. IANA actually lists TCP port 123 for NTP, but unfortunately it doesn't look like there was any demand for it. – user418378 Jun 02 '17 at 01:31