Start service after AWS user-data has run

Question

I am building an AMI based off of an Ubuntu 16.04 AMI.

When I start an instance from my AMI, I would like to pass in a user-data script that runs before the service start up on the AMI.

It looks like the user-data is run in cloud-final.service. If I systemctl status cloud-final or journalctl -u cloud-final I see the output of my user-data script.

I tried to set up my .service file to start the service after cloud-final.service

[Unit]
Description=My Service
After=network.service
After=cloud-final.service

...

but cloud-final has RemainAfterExit=yes, which means it never completes, so my service never starts.

How can I configure AWS Ubuntu to start my service after the user-data script has been run?

If all fails, you could have the script trigger a new target or start your service — eckes, Aug 30 '17 at 20:36

score 11 · Accepted Answer · answered Oct 29 '18 at 16:06

Use

[Unit]
…
After=cloud-final.service
…
[Install]
WantedBy=cloud-init.target

I ran into this as well. journalctl showed user data stuff running after multi-user and before the target cloud-init was reached.

<timestamp> <ip_addr> systemd[1]: Reached target Multi-User System.
[snip]
<timestamp> <ip_addr> systemd[1]: Starting Execute cloud user/final scripts...
<timestamp> <ip_addr> user-data[1592]: my user data stuff
[snip]
<timestamp> <ip_addr> systemd[1]: Started Execute cloud user/final scripts.
[snip]
<timestamp> <ip_addr> systemd[1]: Reached target Cloud-init target.

So I thought to have it be required/wanted by cloud-init instead of the usual multi-user target (which may be what op had), and it just worked.

Diogo Kiss · Answer 2 · 2019-07-10T15:29:50.493

The solution proposed by vlfig works. But, I think that it can be improved.

In my example, I'm trying to run the Puppet agent after cloud-init is finished.

In my case, we wanted Puppet to only run after Cloud Init is done with its set up. In order to achieve that, we need to modify the setup of its systemd unit.

The default execution order for the services and targets that interest us is the following: 1. cloud-init-local.service 2. cloud-init.service 3. cloud-config.target 4. basic.target 5. cloud-config.service 6. puppet.service 7. multi-user.target 8. cloud-final.service 9. cloud-init.target

(stag) dki@appbackend-i-0f0d3f6abd8520f12:~$ systemd-analyze --no-pager critical-chain puppet.service
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

puppet.service @17.737s
└─basic.target @17.291s
  └─paths.target @17.290s
    └─resolvconf-pull-resolved.path @17.290s
      └─sysinit.target @17.282s
        └─cloud-init.service @14.709s +2.572s
          └─systemd-networkd-wait-online.service @13.872s +832ms
            └─systemd-networkd.service @13.487s +377ms
              └─network-pre.target @13.482s
                └─cloud-init-local.service @6.983s +6.495s
                  └─var-lib.mount @11.888s
                    └─local-fs-pre.target @10.584s
                      └─keyboard-setup.service @6.506s +4.072s
                        └─systemd-journald.socket @6.430s
                          └─system.slice @6.416s
                            └─-.slice @3.772s

Dependency graph:

Systemd services dependency graph
Note: The green arrows mean After= and the gray arrows mean Wants=

By having puppet.service in /etc/systemd/system/multi-user.target.wants, systemd automatically creates a dependency order of the type After=puppet.service according to https://www.freedesktop.org/software/systemd/man/systemd.target.html#Default%20Dependencies

Dependency graph:

Puppet service dependency graph original
Note: The green arrows mean After= and the gray arrows mean Wants=

As a consequence, if we try to make it run After=cloud-init.target, it creates a dependency cycle and doesn't start at all.

[Mon Jul  9 12:49:01 2019] systemd[1]: multi-user.target: Found ordering cycle on puppet.service/start
[Mon Jul  9 12:49:01 2019] systemd[1]: multi-user.target: Found dependency on cloud-init.target/start
[Mon Jul  9 12:49:01 2019] systemd[1]: multi-user.target: Found dependency on cloud-final.service/start
[Mon Jul  9 12:49:01 2019] systemd[1]: multi-user.target: Found dependency on multi-user.target/start
[Mon Jul  9 12:49:01 2019] systemd[1]: multi-user.target: Job puppet.service/start deleted to break ordering cycle starting with multi-user.target/start

In order to break this cycle, we need to disable the DefaultDependencies directive. By doing that, we are able to add the After=cloud-init.target to the Puppet unit in order to ensure that it waits for Cloud-Init to be done before it starts running. Both directives are done by creating the /etc/systemd/system/puppet.service.d/override.conf file, which overrides the package's original configurations without changing its original content. The advantage of it is that we can upgrade the package keeping the changes to only the directives we actually need.

Dependency graph:

Puppet service dependency graph afterwards
Note: The green arrows mean After= and the gray arrows mean Wants=

The resulting execution order is: 1. cloud-init-local.service 2. cloud-init.service 3. cloud-config.target 4. basic.target 5. cloud-config.service 6. multi-user.target 7. cloud-final.service 8. cloud-init.target 9. puppet.service

(sand) dki@supplier-integrations-i-035c65e75762aaabd:~$ systemd-analyze --no-pager critical-chain puppet.service
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

puppet.service @43.300s
└─cloud-init.target @43.298s
  └─cloud-final.service @33.272s +10.025s
    └─multi-user.target @33.246s
      └─splunk.service @13.815s +19.429s
        └─basic.target @13.463s
          └─sockets.target @13.463s
            └─docker.socket @13.453s +10ms
              └─sysinit.target @13.436s
                └─cloud-init.service @10.873s +2.557s
                  └─systemd-networkd-wait-online.service @9.182s +1.689s
                    └─systemd-networkd.service @9.034s +143ms
                      └─network-pre.target @9.032s
                        └─cloud-init-local.service @854ms +8.177s
                          └─var-lib.mount @7.587s
                            └─local-fs-pre.target @1.391s
                              └─keyboard-setup.service @569ms +822ms
                                └─systemd-journald.socket @564ms
                                  └─system.slice @564ms
                                    └─-.slice @423ms

Annoyingly on Centos 7 AWS image cloud-init.target does not exist ( and possible not any RH7 based distro ) ! — JamesP, Feb 28 '20 at 13:33
As cloud-init.target appears to just depend on multi-user.target itself using the following override seems to work:[Unit] After=multi-user.target DefaultDependencies=no — JamesP, Feb 28 '20 at 14:27

score 1 · Answer 3 · answered Aug 30 '17 at 20:35

What you could probably do is have the user-data script write a file signifying it is complete (at the end of the script), and have the other service start after somewhere near the end, say, and have it first monitor the file. When it notices the file being created (or timestamp being updated, or contents changed, whatever), then start the rest of the service and exit as normal.

score 0 · Answer 4 · answered Mar 05 '20 at 16:26

After much struggle and thanks to some hints found in this thread, I finally came out a solution that seems to work for my AL2-based AMIs.

My use-case: I have an AMI with an already enabled install.service that starts some scripts to fine tune the app. The end user can use the aws tool to dynamically attach some pre-existing EBS volumes via userdata on launch.

Here is the full install.service file in case it might help someone:

[Unit]
Description=Post launch, pre-installation service
After=cloud-init.target
DefaultDependencies=no
AssertFileIsExecutable=/usr/local/bin/install-script

[Service]
Type=simple
ExecStart=/usr/local/bin/install-script
LimitNOFILE=16384
SuccessExitStatus=0

[Install]
WantedBy=default.target

score 0 · Answer 5 · answered Sep 24 '21 at 14:18

Building on vlfig:

I was trying to do this with pm2. It has a startup script that creates pm2-ubuntu.service so I modified that.

BUT I also found that there was also a symlink inside /etc/systemd/system/multi-user.target.wants that needed to be moved to /etc/systemd/system/cloud-init.target.wants. So there may be other things like that blocking.

Hope that helps.

score 0 · Answer 6 · answered Jan 05 '23 at 05:20

I tried following the advice of the other answers here, but ended up creating dependency cycles.

According to the cloud-init doc's:

For scripts external to cloud-init looking to wait until cloud-init is finished, the cloud-init status --wait subcommand can help block external scripts until cloud-init is done without having to write your own systemd units dependency chains.

score 0 · Answer 7 · answered Jan 05 '23 at 08:14

You can create a new service file for your service and set the After property to cloud-final.service. Then you can use the systemctl enable command to enable your service to start at boot.

For example, create a file /etc/systemd/system/my-service.service with the following contents:

[Unit] Description=My Service After=cloud-final.service

[Service] ExecStart=

[Install] WantedBy=multi-user.target

Then run systemctl enable my-service to enable the service to start at boot.

Start service after AWS user-data has run

7 Answers7