7

Today and yesterday, my server automatically rebooted and failed to bring up the network device during boot. If I reboot the machine again, then it starts up fine, I've also not encountered any issues with this during the past 2 months.

The only error logs I can find relating to this are:

Aug 23 06:37:14 server systemd[1]: Started ifup for ens16.
Aug 23 06:37:14 server systemd[1]: ifup@ens16.service: Main process exited, code=exited, status=1/FAILURE

and

Aug 23 06:37:14 server sh[281]: iptables-restore: line 10 failed
Aug 23 06:37:14 server systemd[1]: ifup@ens16.service: Main process exited, code=exited, status=1/FAILURE
Aug 23 06:37:14 server sh[281]: run-parts: /etc/network/if-pre-up.d/iptables exited with return code 1
Aug 23 06:37:14 server sh[281]: ifup: failed to bring up ens16

/etc/network/if-pre-up.d/iptables contains:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables.up.rules

/etc/iptables.up.rules contains:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [896:90530]
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-port-unreachable
COMMIT

What could possibly be going wrong with this in an intermittent fashion, and how can I make sure it doesn't happen again?

Server Fault
  • 3,714
  • 12
  • 54
  • 89
Sam Bull
  • 325
  • 4
  • 12
  • Start by saving the output of `/sbin/iptables-restore` to a file and see what messages it contains. – RalfFriedl Aug 23 '18 at 17:32
  • Finally happened again. There was no message. The file is empty, after having changed the command to `/sbin/iptables-restore < /etc/iptables.up.rules > /var/log/iptables-output`. – Sam Bull Oct 02 '18 at 13:10
  • 1
    Try `/sbin/iptables-restore < /etc/iptables.up.rules >& /var/log/iptables-output` - ifthere's an error, it's likely to appear on STDERR. NB if your `sh` is really `sh`, not `bash`, modify accordingly. – MadHatter Oct 21 '18 at 11:54
  • 1
    What Linux distribution is this? – Michael Hampton Oct 21 '18 at 13:19
  • @MadHatter Assumed stderr is what was already in the error logs shown above. But, will try that out. – Sam Bull Oct 21 '18 at 15:09
  • @MichaelHampton Debian stable. – Sam Bull Oct 21 '18 at 15:09
  • @MadHatter I would modify that command to include the PID in the file name like this: `/sbin/iptables-restore < /etc/iptables.up.rules >& /var/log/iptables-output.$$`. I have a suspicion that script gets executed more than once during boot, it would be nice to have the output from all the invocations. – kasperd Oct 22 '18 at 15:12
  • try appending instead of overwriting... i.e. restore < /etc/iptables.up.rules 2>&1 >> /var/log/iptables-output.$$ – TheCompWiz Oct 22 '18 at 19:14
  • @SamBull Please, could you provide output of command `systemctl list-units --all 'ifup*.service'`? I believe the network interface fails to start due to a concurrent execution of your `iptables` script. – Anderson Medeiros Gomes Oct 23 '18 at 00:48
  • @AndersonM.Gomes This is on a normal successful boot: `UNIT LOAD ACTIVE SUB DESCRIPTION ifup@ens16.service loaded active exited ifup for ens16 1 loaded units listed.` – Sam Bull Oct 23 '18 at 10:39

1 Answers1

4

I still suspect that two executions of /etc/network/if-pre-up.d/iptables are running at the same time throughout the boot process. Because of systemd normal behavior of starting things concurrently unless advised not to do so, I believe the boot process triggers one script process for the lo interface and another for the ens16 interface. That would result in a concurrent execution of iptables-restore, which may cause errors such as iptables-restore: line 10 failed. I am unable to supply evidences though.

I am used to managing CentOS and Red Hat systems. Once upon a time, one of such servers failed to initialize iptables service on boot because systemd was starting ip6tables concurrently. That specific error is documented here: https://bugzilla.redhat.com/show_bug.cgi?id=1477413

I suggest you to handle concurrency in your script, for example, by using flock:

#!/bin/sh
/usr/bin/flock /run/.iptables-restore /sbin/iptables-restore < /etc/iptables.up.rules

Alternatively, you could check the actual value of ${IFACE} variable before restoring iptables rules (reference: man 5 interfaces):

#!/bin/sh
if [ "${IFACE}" == ens16 ]; then
    /sbin/iptables-restore < /etc/iptables.up.rules
fi

Additionally, if you just want to load iptables rules at boot time, I suggest you to use iptables-persistent instead:

# apt-get install iptables-persistent netfilter-persistent
# mv -v /etc/iptables.up.rules /etc/iptables/rules.v4
# systemctl enable netfilter-persistent.service
# rm -v /etc/network/if-pre-up.d/iptables
Sam Bull
  • 325
  • 4
  • 12
Anderson Medeiros Gomes
  • 2,477
  • 9
  • 20
  • I added an `echo "${IFACE}"` to the script and rebooted, and it seems that it is in fact running 3 times, for `--all`, `ens16` and `lo`. So, I'll try your suggestions, and hope that fixes the problem. – Sam Bull Oct 24 '18 at 12:28
  • 1
    Just to follow up on this 1.5 years later. I've never seen the issue happen since, so I'm confident this answer was the correct solution. – Sam Bull Mar 01 '20 at 11:19