How to track the progress of unattended-upgrades or cloud-init?

Question

I am looking to use Packer (builds images) and Ansible (provisioner) to provision an Ubuntu AMI.

"name": "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*",

I am having difficulties because some of the tasks try to install packages using apt, but the lock is held by another process. I am having difficulty identifying what processes are holding the lock most importantly, what is the progress with that certain process.

By default the AMIs that amazon sets up will install security updates on launch [0], so I assume this is it. As the docs explain, it might be related to cloud-init? I believe that is also related to unattended-upgrades, since as you can see in this [1] paste, there is an unattended-upgrades-shutdown process that is waiting for some other process (apt?) to finish installing upgrades before shutdown.

If I use sudo lslocks, I get

    amazon-ebs:         "COMMAND           PID  TYPE SIZE MODE  M START END PATH",
    amazon-ebs:         "lvmetad           433 POSIX   4B WRITE 0     0   0 /run/lvmetad.pid",
    amazon-ebs:         "iscsid           1082 POSIX   5B WRITE 0     0   0 /run/iscsid.pid",
    amazon-ebs:         "lxcfs            1110 POSIX   5B WRITE 0     0   0 /run/lxcfs.pid",
    amazon-ebs:         "cron             1134 FLOCK   5B WRITE 0     0   0 /run/crond.pid",
    amazon-ebs:         "atd              1127 POSIX   5B WRITE 0     0   0 /run/atd.pid"

which is not telling me much about all the files that are locks that I would be interested in.

If I tail /var/log/cloud-init-output.log, I see that cloud-init is done working.

If I tail /var/log/dpkg.log, I see logs from September 13 which is not today.

If I tail /var/log/apt/term.log, I see logs from September 13 which is not today.

This

>&1 sudo fuser '/var/lib/dpkg/lock-frontend' || echo aa ;
>&1 sudo fuser -vvv /var/lib/apt/lock || echo a ;
>&1 sudo lsof /var/lib/apt/lists/lock || echo b ;
>&1 sudo lsof /var/lib/dpkg/lock || echo c ;
>&1 sudo lsof /var/cache/apt/archives/lock || echo d ;

is outputting

aa
a
b
c
d

so I understand that these lock files do not exist. I'm puzzled, because there is an error about a lock file: Failed to lock apt for exclusive operation .

How could I find what lock file that is. And most importantly, how could I track the progress of the process holding that lock?

Thank you!

[0:security updates docs] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-linux-ami-basics.html

[1:ps paste] https://pastebin.ubuntu.com/p/JGNkfVFHGJ/

See this Unix exchange answer, https://unix.stackexchange.com/a/315517 — Rickard von Essen, Oct 08 '19 at 04:22

Chad Smith · Answer 1 · 2019-10-07T19:37:57.913

1

As of cloud-init v.18.2 or later, cloud-init status --wait will block until cloud-init is finished running. So it's an easy 'hook' for a script to leverage before doing the rest of its work.

edited Oct 07 '19 at 19:37

answered Oct 07 '19 at 19:29

Chad Smith

63
4

Thank you Chad. It appears that even after cloud-init is done working, I am not able to hold the apt lock. – Slackware Oct 07 '19 at 21:33

score 0 · Answer 2 · answered Oct 15 '19 at 23:18

It seems like using Packer with Ansible makes things a little bit complicated. For some reason, I had set, in my Packer configurations, that the EC2 would be ssh'ed into using the ubuntu user and that Ansible would be using the root user. This caused Ansible to NOT try to run "sudo" because it thought it was already root. Thus, it was not able to get a hold of the lock in order to install packages.

However, this does not answer how to track the packages installation progress. I think looking at the cloud-init gave a nice view of what was happening. Cloud-init was done working. The apt logs showed no progress, so I believe that it was not installing anything.

How to track the progress of unattended-upgrades or cloud-init?

2 Answers2