AWS instance Stop and Terminate don't execute systemd stop script

Question

I have a shutdown script that executes just fine after cli reboot or shutdown, but never executes when an instance is stopped or terminated from aws console.

[Unit]
Description=Gracefully shut down remnode to avoid database dirty flag
DefaultDependencies=no
After=poweroff.target shutdown.target reboot.target halt.target kexec.target
RequiresMountsFor=/data
Requires=network-online.target network.target data.mount

[Service]
Type=oneshot
ExecStop=/root/node_shutdown.sh
RemainAfterExit=yes
KillMode=none

[Install]
WantedBy=multi-user.target

What am I missing?

EDIT

Here is the journalctl log for my service. The first two entries are reboot via cli, the last 3 entries are 2 instance stops from AWS console. As you can see, there's not even a mention of winding down my service during instance stop. Rebooting from cli however outputs my echo and system logs startup as well as shutdown of the service.

-- Logs begin at Sun 2019-10-13 13:02:54 UTC, end at Mon 2019-10-14 19:41:01 UTC. --
Oct 13 13:03:24 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
Oct 14 19:27:23 ip-10-0-1-182 systemd[1]: Stopping Gracefully shut down remnode to avoid database dirty flag...
Oct 14 19:27:23 ip-10-0-1-182 node_shutdown.sh[10635]: RUNNING SHUTDOWN SCRIPT
Oct 14 19:27:23 ip-10-0-1-182 systemd[1]: Stopped Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:27:37 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
Oct 14 19:28:17 ip-10-0-1-182 systemd[1]: Stopping Gracefully shut down remnode to avoid database dirty flag...
Oct 14 19:28:17 ip-10-0-1-182 node_shutdown.sh[1712]: RUNNING SHUTDOWN SCRIPT
Oct 14 19:28:18 ip-10-0-1-182 systemd[1]: Stopped Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:28:32 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:34:05 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.
-- Reboot --
Oct 14 19:40:26 ip-10-0-1-182 systemd[1]: Started Gracefully shut down remnode to avoid database dirty flag.

EDIT 2:

Because it seems like shutdown trigger is basically never pulled, I've also tried setting kernel sysctl kernel.poweroff_cmd to /bin/systemctl poweroff following this thread. Hoping it would trigger the even, but no luck.

have you tried to put `RemainAfterExit` before `ExecStop`? also can you confirm it is executing during script execution or even not calling at all? — asktyagi, Oct 14 '19 at 16:26
@asktyagi I have tried various orders, they don't seem to have any impact. Also, I added logs to my post. As you can see, it doesn't even seem to log shutdown of my process. — Serg Chernata, Oct 14 '19 at 19:44
I'll update my post when I have more information, but after reaching out to AWS support, the answer seems to be that their primary, canonical Ubuntu 18.04 AMI is the reason for this bug. Using a different AMI, like .NET edition, fixes the issue. — Serg Chernata, Oct 18 '19 at 16:29
If You are relying on systemd-logind to initiate shutdown/poweroff on power button press as usual, there can be applications that acquire some inhibtor-lock on shutdown/poweroff event handling of systemd which may cause systemd to ignore or delay this events, leading to EC2 instance externally powered off after timeout of 4 minutes. — EOhm, Oct 20 '19 at 09:30
You could try ewith wetting `PowerKeyIgnoreInhibited=yes` in /etc/logind.conf and restart systemd-logind to ensure that no application can inhibit systemd-logind from powering off after power button press using inhibitor. — EOhm, Oct 20 '19 at 09:36

Adam Collyer · Answer 1 · 2019-10-18T00:44:00.157

Just a hunch, but might be worth exploring:

Have you tried using Before= with ExecStart= instead of After= with ExecStop=? i.e. start the node_shutdown.sh script before the final state of a shutdown/halt/reboot is reached. There are answers to similar questions here and here that allude to this.

That said, the accepted answer for the first linked question uses neither option! And why it works on a manual shutdown but not a console shutdown I'm not sure. I know a console shutdown registers as a "power button pressed" event - maybe this event is treated differently by the OS?

A power button press triggers an ACPI event which (I think) runs /etc/acpi/events/hibinit-power => /etc/acpi/actions/hibinit-power.sh, which itself sends a poweroff command to the message bus. It might be worth checking the contents of that .sh in case there's anything funky in there. My vanilla Ubuntu 18.04 looks like this:

#!/bin/sh

# shut down system in a way that respects inhibitors
# see: https://github.com/systemd/systemd/pull/9356
dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager.PowerOff boolean:false

As asktyagi suggests it's probably worth determining if the script is actually running - maybe try writing to a log at the top of the node_shutdown script?

Edit - here are the contents of the following files:

/etc/acpi/events/hibinit-power

event=button/power
action=/etc/acpi/actions/hibinit-power.sh "%e"

/etc/acpi/events/hibinit-sleep

# ACPID config to power down machine if powerbutton is pressed, but only if
# no gnome-power-manager is running

event=button/sleep.*
action=/etc/acpi/actions/sleep.sh %e

/etc/acpi/actions/sleep.sh

#!/bin/sh

#PATH=/sbin:/bin:/usr/bin

do_hibernate() {
    if [ -d /run/systemd/system ]; then
        systemctl hibernate
    else
        pm-hibernate
        swapoff /swap-hibinit
    fi
}


case "$2" in
    SBTN)
        swapon /swap-hibinit && do_hibernate
        ;;
    *)
        logger "ACPI action undefined: $2" ;;
esac

One thing to also check is /etc/systemd/logind.conf to see how the system handles the power key being pressed. Mine appears to have all the defaults set, notably the ones below:

[Login]
...
#HandlePowerKey=poweroff
#HandleSuspendKey=suspend
#HandleHibernateKey=hibernate
...
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no

I double checked this with a fresh Ubuntu instance built from the AWS stock image (ami-02df9ea15c1778c9c in EU-West-1)

I did try using `Before` configuration, unfortunately. And I am writing to log, see the last portion of my question. The second part of your answer is interesting. I'm also using Ubuntu 18.04 but my files are different from yours. I have `hibinit-sleep` instead of `hibinit-power`. — Serg Chernata, Oct 17 '19 at 11:58
Can you include your contents of `/etc/acpi/events/hibinit-power` ? — Serg Chernata, Oct 17 '19 at 14:07
But You should note the acpi scripts are only used from acpid service, not for systemd. So it needs to be started to handle them. — EOhm, Oct 20 '19 at 09:26

score 0 · Answer 2 · answered Oct 19 '19 at 20:39

I had exactly the same problem on Ubuntu 18.04 AMI. I read the above article and tried updating packages.

> sudo apt update
> sudo apt upgrade

I don't know why, but it works fine now even when EC2 instance is shutdown via web console. And there are new acpi related scripts that were missing(hibinit-power, hibinit-power.sh). Would you like to try it out?

AWS instance Stop and Terminate don't execute systemd stop script

2 Answers2