Run Systemd Service Unit After AWS EBS Volume Mount

Question

I launch m5.large (nitro-based) EC2 instance from Ubuntu AMI and attach EBS volume. There is systemd as a default init system. As AWS documentation "Making an Amazon EBS Volume Available for Use on Linux" stands, I mount EBS volume within user data:

#!/bin/bash

# Sleep gives the SSD drive a chance to mount before the user data script completes.
sleep 15

mkdir /application

mount /dev/nvme1n1 /application

I need Nginx and provide site configuration for it at EBS volume. For default nginx package with systemd unit file I declare a dependency on the mount with RequiresMountsFor directive within drop-in:

# /lib/systemd/system/nginx.service

[Unit]
Description=A high performance web server and a reverse proxy server
After=network.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/nginx.service.d/override.conf

[Unit]
RequiresMountsFor=/application

[Service]
Restart=always

But this doesn't help to run Nginx only after mount will be completed (in user data) for some reason. I can see the mount unit for /application path, but I don't see Required=application.mount as I'd expect:

$ sudo systemctl show -p After,Requires nginx
Requires=system.slice sysinit.target -.mount
After=sysinit.target -.mount systemd-journald.socket basic.target application.mount system.slice network.target

Nginx service still tries to run before cloud-init completes user data execution, exhausts all attempts to run the service and fails:

Apr 08 15:34:32 hostname nginx[1303]: nginx: [emerg] open() "/application/libexec/etc/nginx/nginx.site.conf" failed (2: No such file or directory) in /etc/nginx/sites-e
Apr 08 15:34:32 hostname nginx[1303]: nginx: configuration file /etc/nginx/nginx.conf test failed
Apr 08 15:34:32 hostname systemd[1]: nginx.service: Control process exited, code=exited status=1
Apr 08 15:34:32 hostname systemd[1]: Failed to start A high performance web server and a reverse proxy server.

I assume the service should be started by systemd on mount notification for the specified path /application. What am I missing?

What is the most flexible and correct way to mount EBS volumes at Ubuntu + systemd?

Answer 1

Here is my take on a solution that attempts to honour the constraints and limitations mentioned in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html#identify-nvme-ebs-device. Not currently used in production but that's the aim...

NVMe device names follow the pattern /dev/nvmen, where is the enumeration order, and, for EBS, is 1. Occasionally, devices can respond to discovery in a different order in subsequent instance starts, which causes the device name to change.

The basic idea is to mimic the systemd Requires=foo.mount behaviour by wrapping the EBS mount procedure in a systemd service. Other services that depend on the EBS mount can simply specify that they must be started After the mount service is available.

The functionality builds upon the udev module that sym-links the physical device with the requested device specified when the volume is attached. For example /dev/sdf links to /dev/nvme...). See https://github.com/oogali/ebs-automatic-nvme-mapping and the above guide for more details.

The data-mount.service waits in a sleep loop for the sym-link to appear (allowing you time to attach the volume), then mounts the volume at the defined mount point, formatting it if necessary.

Service to mount the EBS volume

/sbin/ec2-boot-mount-ebs-volume

#!/bin/bash
#
# Copyright 2019 - binx.io B.V.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Based on https://github.com/binxio/ec2-boot-mount-ebs-volume.
#
# Requires the udev rule that automatically creates a sym-link to the actual device. See the note on udev on https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html#identify-nvme-ebs-device

set -e -o pipefail
exec 1> >(logger -s -t $(basename $0)) 2>&1

function wait_for_device {
        while [[ ! -b $(readlink -f $1) ]]; do
            echo "waiting for device $1" >&2
            sleep 5;
        done
}

function label_device {
        label=$(blkid $1 | sed -e 's/.*LABEL="\([^"]*\)".*/\1/')
        if [[ -z $label ]]; then
                echo "INFO: labeling $1 with $2">&2
                e2label $1 $2;
        elif [[ $label != $2 ]]; then
                echo "ERROR: device $1 already has label $label, expected $2">&2; exit 1
        fi
}

function main {
        local device mount_point fstype options
        if [[ $# -ne 4 ]] ;then
                echo "Usage: $(basename $0) device-name mount-point fstype options" >&2
                echo "  waits for the volume, formats it if unformatted and mounts it." >&2
                exit 1
        fi
        device="$1"
        mount_point="$2"
        fstype="$3"
        options="$4"

        wait_for_device $device
        real_device=$(readlink -f $device)
        if [[ $real_device != $device ]] ;then
                echo "INFO: $device appeared as $real_device" >&2
        fi

        if grep -q "^$real_device $mount_point " /proc/mounts; then
                echo "INFO: $real_device already mounted on $mount_point" >&2
                return 0
        fi

        if [[ -z $(blkid $real_device) ]] ; then
                echo "INFO: formatting $real_device" >&2
                mkfs -L $mount_point -t $fstype $real_device
        else
                echo "INFO: $real_device already formatted" >&2
                label_device $real_device $mount_point
        fi

        echo "INFO: mounting $real_device on $mount_point" >&2
        mkdir -p $mount_point
        mount -t $fstype -o "$options" $real_device $mount_point
}

main "$@"

/etc/systemd/system/data-mount.service

[Unit]
Description=Mount Data Volume
After=cloud-init-local.service

[Service]
Type=oneshot
RemainAfterExit=yes
# /dev/sdf should be replaced with the device name you requested when attaching the volume
ExecStart=/sbin/ec2-boot-mount-ebs-volume /dev/sdf /mnt/data ext4 "defaults"
ExecStop=umount -v /mnt/data

[Install]

Other Services

/etc/systemd/system/other-service.service.d/override.conf

[Unit]
# if the data-mount.service stops then this service also needs to stop
BindsTo=data-mount.service
# ensure this service starts after the mount is available
After=data-mount.service