13

I've got a couple services, most notably nginx and ntpd, that depend on having working DNS resolution to start properly. Right now neither of these services start correctly at boot time, but do start fine when manually intervened on once the machine is up, with some messages in the logs about not being able to resolve names.

This leads me to believe I'm having a race condition with systemd. My servers point to 127.0.0.1 for their nameservers. Binded to localhost:53 is pdns-recursor. I've set ntp and nginx to be WantedBy pdns-recursor in their unit files as follows

[Unit]
WantedBy=pdns-recursor.service

However I still receive log messages in both nginx and ntp about failing to resolve names at boot time.

How can I verify that DNS is completely up before these services attempt to start? I am using Ubuntu 16.04

Aug 09 22:35:25 host.blah ntpd[3574]: restrict: ignoring line 21, address/host 'ntp.blah' unusable.
Aug 09 22:35:26 host.blah ntpd[3574]: restrict: ignoring line 23, address/host 'ntp.blah' unusable.
Aug 09 22:35:28 host.blah ntpd[3574]: restrict: ignoring line 25, address/host 'ntp.blah' unusable.
Aug 09 22:35:29 host.blah ntpd[3574]: restrict: ignoring line 27, address/host 'ntp.blah' unusable.
Brando__
  • 412
  • 1
  • 3
  • 11

5 Answers5

17

I placed an ExecStartPre in my ntp and nginx unit files to continue trying to resolve a name before continuing on.

This works properly. Normal systemd constructions like waiting for other services to be online are unreliable as just because the pdns daemon is up doesn't mean its necessarily able to answer queries in the first few milliseconds of being up.

ExecStartPre=/bin/sh -c 'until host example.com; do sleep 1; done'
fission
  • 3,601
  • 2
  • 21
  • 31
Brando__
  • 412
  • 1
  • 3
  • 11
  • Brilliant! I have a host with unusual DNS that sometimes starts later; having an until host specific for that hosts solves the problem. Thanks! – Jamie S Nov 26 '22 at 00:03
7

Try using:

[Unit]
After=network-online.target
Wants=network-online.target

There's a full write-up on Unix & Linux as well as on the FreeDesktop site.

jeberle
  • 295
  • 2
  • 3
  • The link to Unix & Linux is a good source of explanation. – rodvlopes Oct 28 '19 at 03:09
  • 2
    This doesn't work reliably as "What precisely this requires is left to the implementation of the network managing service." and certainly in Raspberry Pi OS (Buster) the resolver just has to be running, not connected to a possibly not-yet-running DNS server. Eg in the case of a power-outage recovery. This isn't a big deal for the OP but is a real issue if you need a functional DNS. – lbt Jul 25 '21 at 11:38
  • I does not work in my case with Postfix. – Maxim Dec 25 '22 at 11:31
6

systemd is flexible enough to do this:

[Unit]
Description=Wait for DNS to come up using 'host'
After=nss-lookup.target

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'until host yourhost.com; do sleep 1; done'

[Install]
WantedBy=multi-user.target

Then you can depend on this unit in your unit file that requires dns to mount:

[Unit]
Description=mount nfs
After=dns-ready.service
mianos
  • 215
  • 2
  • 7
3

So, I have the same dilemma with a remote mount cifs drive that needs to be mounted via DNS FQDN rather than IP. I tried a batch of things, but so far I have to agree with Brando's solution (short of writing an entire Systemd service and pulling it in before the .mount or network-online.target).

The only alternative to Brando's that I have found (that actually works in practice, rather than just in theory from the man pages) is to swat the fly with a sledge hammer, and put an ExecStartPre= into systemd-networkd-wait-online.service:

[Service]
...
ExecStartPre=/bin/sleep 15

If you are keen, you can also append to the ExecStart= using --interface=interface_name (and optionally --timeout=). This helps because I have a vlan sub interface, but wait-online was just tracking the first of any interfaces (loop back excluded) that would come online. By itself though, it didn't fix the problem.

I though using -o _netdev within the mount definition of /etc/fstab would fix it (and in theory in Systemd remote drives wait for the network-online.target by default), but alas, no dice. Listing the dependencies of the resultant mnt-remote.mount file does show that network-online.target is a precursor, but DNS still failed. Manually running the service post boot works fine, I'm just getting screwed by some sort of race condition between the target coming up and DNS actually resolving prior to the mount process happening. That, and the fact that the definition of an 'online' interface varies from person to person and case to case (my use case requires DNS, others obviously don't - a topic that is covered extensively on FreeDesktop: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget).

Brando's solution is more elegant. This one also needs trial an error to minimise the sleep time but still have it work reliably, subject to how long it takes your DHCP/DNS/NIC to get all sorted out.

I throw this out there as an inferior solution just in case Brando's doesn't work for you for some reason.

Another work around would be to punch an entry into the local hosts file, but I wasn't interested in doing that for a case where the IP to FQDN changes on a relatively frequent basis (similar issue with just mounting it using the IP in the first place).

Pang
  • 273
  • 3
  • 8
Chris
  • 181
  • 1
  • 6
0

I agree with your race condition diagnosis...unless dependencies are explicitly configured, systemd will be trying to start these services in parallel. But I don't think the WantedBy directive is going to help you, since that affects installation dependencies, not start-up dependencies, per this man page.

I think what you're looking for is a combination of the Wants directive and the After directive. For each of ntp and nginx, I think you want to add the following to your unit file:

Wants=pdns-recursor.service
After=pdns-recursor.service

These two options should ensure that the DNS service is started before the ntp/nginx service, which should hopefully solve your race.

Being a belt-and-suspenders kind of guy, I'd actually recommend putting your name/IP mapping in your /etc/hosts file; that way the other services can start even if your DNS fails. If you really want to try to use DNS in preference to static files, you can switch the resolution order.

Jeremy Dover
  • 318
  • 1
  • 6