4

This morning i was alerted to the fact that both apache instances serving my app were not responding to requests from my load balancer. I attempted apachectl restart and it said apache was not running. So, i started apache on both instances and got the service up again. I then followed the logs and worked out that both had performed upgrades via the unattended-upgrades package moments before they stopped responding.

/var/log/unattended-upgrades/unattended-upgrades.log

2013-07-02 06:30:51,875 INFO Starting unattended upgrades script
2013-07-02 06:30:51,875 INFO Allowed origins are: ['o=Ubuntu,a=precise-security']
2013-07-02 06:33:57,771 INFO Packages that are upgraded: accountsservice apache2 apache2-mpm-prefork apache2-utils apache2.2-bin apache2.2-common apparmor apport apt apt-transport-https apt-utils bind9-host binutils dbus dnsutils gnupg gpgv isc-dhcp-client isc-dhcp-common krb5-locales libaccountsservice0 libapt-inst1.4 libapt-pkg4.12 libbind9-80 libc-bin libc-dev-bin libc6 libc6-dev libcurl3-gnutls libdbus-1-3 libdbus-glib-1-2 libdns81 libdrm-intel1 libdrm-nouveau1a libdrm-radeon1 libdrm2 libexpat1 libfreetype6 libgc1c2 libgnutls-dev libgnutls-openssl27 libgnutls26 libgnutlsxx27 libisc83 libisccc80 libisccfg82 liblwres80 libruby1.8 libx11-6 libx11-data libxcb1 libxext6 libxml2 linux-firmware linux-image-virtual linux-libc-dev linux-virtual multiarch-support openssl perl perl-base perl-modules python-apport python-crypto python-keyring python-problem-report python-software-properties ri1.8 ruby1.8 ruby1.8-dev sudo tzdata update-manager-core
2013-07-02 06:33:57,772 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg_2013-07-02_06:33:57.772399.log'
2013-07-02 06:36:10,584 INFO All upgrades installed

I'm running Ubuntu 12.04 on Amazon EC2 servers. I have unattended-upgrades installed and configured as follows:

/etc/apt/apt.conf.d/50unattended-upgrades

// Automatically upgrade packages from these (origin:archive) pairs
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
//  "${distro_id}:${distro_codename}-updates";
//  "${distro_id}:${distro_codename}-proposed";
//  "${distro_id}:${distro_codename}-backports";
};

// List of packages to not update
Unattended-Upgrade::Package-Blacklist {
};

/etc/apt/apt.conf.d/20auto-upgrades

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

I've struggled to find documentation about what happens to running processes during an upgrade.

- Is this expected behaviour? Or should unattended-upgrades restart apache after upgrading it? - What can I do to ensure apache is restarted correctly? Should I just blacklist the apache package?

EDIT:

The upgrade log contains the following extract:

/var/log/unattended-upgrades/unattended-upgrades-dpkg_2013-07-02_06:33:57.772399.log

Preparing to replace apache2 2.2.22-1ubuntu1 (using .../apache2_2.2.22-1ubuntu1.3_amd64.deb) ...
Unpacking replacement apache2 ...
Preparing to replace apache2-mpm-prefork 2.2.22-1ubuntu1 (using .../apache2-mpm-prefork_2.2.22-1ubuntu1.3_amd64.deb) ...
* Stopping web server apache2
[Tue Jul 02 06:34:49 2013] [warn] NameVirtualHost *:80 has no VirtualHosts
 ... waiting    ...done.

This is the full upgrade log:

Extracting templates from packages: 100%
Preconfiguring packages ...
(Reading database ... 47409 files and directories currently installed.)
Preparing to replace perl 5.14.2-6ubuntu2 (using .../perl_5.14.2-6ubuntu2.3_amd64.deb) ...
Unpacking replacement perl ...
Preparing to replace perl-base 5.14.2-6ubuntu2 (using .../perl-base_5.14.2-6ubuntu2.3_amd64.deb) ...
Unpacking replacement perl-base ...
Processing triggers for man-db ...
Setting up perl-base (5.14.2-6ubuntu2.3) ...
(Reading database ... 47409 files and directories currently installed.)
Preparing to replace perl-modules 5.14.2-6ubuntu2 (using .../perl-modules_5.14.2-6ubuntu2.3_all.deb) ...
Unpacking replacement perl-modules ...
Preparing to replace libc-dev-bin 2.15-0ubuntu10 (using .../libc-dev-bin_2.15-0ubuntu10.2_amd64.deb) ...
Unpacking replacement libc-dev-bin ...
Preparing to replace libc6-dev 2.15-0ubuntu10 (using .../libc6-dev_2.15-0ubuntu10.2_amd64.deb) ...
Unpacking replacement libc6-dev ...
Preparing to replace libc-bin 2.15-0ubuntu10 (using .../libc-bin_2.15-0ubuntu10.2_amd64.deb) ...
Unpacking replacement libc-bin ...
Processing triggers for man-db ...
Setting up libc-bin (2.15-0ubuntu10.2) ...
(Reading database ... 47409 files and directories currently installed.)
Preparing to replace libc6 2.15-0ubuntu10 (using .../libc6_2.15-0ubuntu10.2_amd64.deb) ...
Unpacking replacement libc6 ...
Setting up libc6 (2.15-0ubuntu10.2) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
(Reading database ... 47409 files and directories currently installed.)
Preparing to replace linux-libc-dev 3.2.0-29.46 (using .../linux-libc-dev_3.2.0-48.74_amd64.deb) ...
Unpacking replacement linux-libc-dev ...
Preparing to replace tzdata 2012b-1 (using .../tzdata_2012e-0ubuntu0.12.04.1_all.deb) ...
Unpacking replacement tzdata ...
Setting up tzdata (2012e-0ubuntu0.12.04.1) ...

Current default time zone: 'Etc/UTC'
Local time is now:      Tue Jul  2 06:34:23 UTC 2013.
Universal Time is now:  Tue Jul  2 06:34:23 UTC 2013.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

(Reading database ... 47409 files and directories currently installed.)
Preparing to replace libapt-pkg4.12 0.8.16~exp12ubuntu10 (using .../libapt-pkg4.12_0.8.16~exp12ubuntu10.10_amd64.deb) ...
Unpacking replacement libapt-pkg4.12 ...
Setting up libapt-pkg4.12 (0.8.16~exp12ubuntu10.10) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
(Reading database ... 47409 files and directories currently installed.)
Preparing to replace gpgv 1.4.11-3ubuntu2 (using .../gpgv_1.4.11-3ubuntu2.2_amd64.deb) ...
Unpacking replacement gpgv ...
Processing triggers for man-db ...
Setting up gpgv (1.4.11-3ubuntu2.2) ...
(Reading database ... 47409 files and directories currently installed.)
Preparing to replace gnupg 1.4.11-3ubuntu2 (using .../gnupg_1.4.11-3ubuntu2.2_amd64.deb) ...
Unpacking replacement gnupg ...
Processing triggers for install-info ...
Processing triggers for man-db ...
Setting up gnupg (1.4.11-3ubuntu2.2) ...
(Reading database ... 47409 files and directories currently installed.)
Preparing to replace apt 0.8.16~exp12ubuntu10 (using .../apt_0.8.16~exp12ubuntu10.10_amd64.deb) ...
Unpacking replacement apt ...
Processing triggers for man-db ...
Setting up apt (0.8.16~exp12ubuntu10.10) ...
Installing new version of config file /etc/apt/apt.conf.d/01autoremove ...
gpg: key 437D05B5: "Ubuntu Archive Automatic Signing Key <ftpmaster@ubuntu.com>" not changed
gpg: key FBB75451: "Ubuntu CD Image Automatic Signing Key <cdimage@ubuntu.com>" not changed
gpg: Total number processed: 2
gpg:              unchanged: 2
(Reading database ... 47410 files and directories currently installed.)
Preparing to replace libapt-inst1.4 0.8.16~exp12ubuntu10 (using .../libapt-inst1.4_0.8.16~exp12ubuntu10.10_amd64.deb) ...
Unpacking replacement libapt-inst1.4 ...
Preparing to replace libdbus-1-3 1.4.18-1ubuntu1 (using .../libdbus-1-3_1.4.18-1ubuntu1.4_amd64.deb) ...
Unpacking replacement libdbus-1-3 ...
Preparing to replace libdrm2 2.4.32-1ubuntu1 (using .../libdrm2_2.4.39-0ubuntu0.2_amd64.deb) ...
Unpacking replacement libdrm2 ...
Preparing to replace libdrm-intel1 2.4.32-1ubuntu1 (using .../libdrm-intel1_2.4.39-0ubuntu0.2_amd64.deb) ...
Unpacking replacement libdrm-intel1 ...
Preparing to replace libdrm-nouveau1a 2.4.32-1ubuntu1 (using .../libdrm-nouveau1a_2.4.39-0ubuntu0.2_amd64.deb) ...
Unpacking replacement libdrm-nouveau1a ...
Preparing to replace libdrm-radeon1 2.4.32-1ubuntu1 (using .../libdrm-radeon1_2.4.39-0ubuntu0.2_amd64.deb) ...
Unpacking replacement libdrm-radeon1 ...
Preparing to replace apparmor 2.7.102-0ubuntu3 (using .../apparmor_2.7.102-0ubuntu3.7_amd64.deb) ...
Unpacking replacement apparmor ...
Preparing to replace libgnutls-dev 2.12.14-5ubuntu3.3 (using .../libgnutls-dev_2.12.14-5ubuntu3.4_amd64.deb) ...
Unpacking replacement libgnutls-dev ...
Preparing to replace libgnutlsxx27 2.12.14-5ubuntu3.3 (using .../libgnutlsxx27_2.12.14-5ubuntu3.4_amd64.deb) ...
Unpacking replacement libgnutlsxx27 ...
Preparing to replace libgnutls-openssl27 2.12.14-5ubuntu3.3 (using .../libgnutls-openssl27_2.12.14-5ubuntu3.4_amd64.deb) ...
Unpacking replacement libgnutls-openssl27 ...
Preparing to replace libgnutls26 2.12.14-5ubuntu3.3 (using .../libgnutls26_2.12.14-5ubuntu3.4_amd64.deb) ...
Unpacking replacement libgnutls26 ...
Preparing to replace libcurl3-gnutls 7.22.0-3ubuntu4 (using .../libcurl3-gnutls_7.22.0-3ubuntu4.1_amd64.deb) ...
Unpacking replacement libcurl3-gnutls ...
Preparing to replace libdbus-glib-1-2 0.98-1ubuntu1 (using .../libdbus-glib-1-2_0.98-1ubuntu1.1_amd64.deb) ...
Unpacking replacement libdbus-glib-1-2 ...
Preparing to replace libx11-data 2:1.4.99.1-0ubuntu2 (using .../libx11-data_2%3a1.4.99.1-0ubuntu2.1_all.deb) ...
Unpacking replacement libx11-data ...
Preparing to replace libxcb1 1.8.1-1 (using .../libxcb1_1.8.1-1ubuntu0.2_amd64.deb) ...
Unpacking replacement libxcb1 ...
Preparing to replace libx11-6 2:1.4.99.1-0ubuntu2 (using .../libx11-6_2%3a1.4.99.1-0ubuntu2.1_amd64.deb) ...
Unpacking replacement libx11-6 ...
Preparing to replace libxext6 2:1.3.0-3build1 (using .../libxext6_2%3a1.3.0-3ubuntu0.1_amd64.deb) ...
Unpacking replacement libxext6 ...
Preparing to replace libxml2 2.7.8.dfsg-5.1ubuntu4 (using .../libxml2_2.7.8.dfsg-5.1ubuntu4.4_amd64.deb) ...
Unpacking replacement libxml2 ...
Preparing to replace libexpat1 2.0.1-7.2ubuntu1 (using .../libexpat1_2.0.1-7.2ubuntu1.1_amd64.deb) ...
Unpacking replacement libexpat1 ...
Preparing to replace libfreetype6 2.4.8-1ubuntu2 (using .../libfreetype6_2.4.8-1ubuntu2.1_amd64.deb) ...
Unpacking replacement libfreetype6 ...
Selecting previously unselected package linux-image-3.2.0-48-virtual.
Unpacking linux-image-3.2.0-48-virtual (from .../linux-image-3.2.0-48-virtual_3.2.0-48.74_amd64.deb) ...
Done.
Preparing to replace multiarch-support 2.15-0ubuntu10 (using .../multiarch-support_2.15-0ubuntu10.2_amd64.deb) ...
Unpacking replacement multiarch-support ...
Processing triggers for ureadahead ...
Processing triggers for man-db ...
Setting up multiarch-support (2.15-0ubuntu10.2) ...
(Reading database ... 48302 files and directories currently installed.)
Preparing to replace apt-utils 0.8.16~exp12ubuntu10 (using .../apt-utils_0.8.16~exp12ubuntu10.10_amd64.deb) ...
Unpacking replacement apt-utils ...
Preparing to replace isc-dhcp-client 4.1.ESV-R4-0ubuntu5 (using .../isc-dhcp-client_4.1.ESV-R4-0ubuntu5.5_amd64.deb) ...
Unpacking replacement isc-dhcp-client ...
Preparing to replace isc-dhcp-common 4.1.ESV-R4-0ubuntu5 (using .../isc-dhcp-common_4.1.ESV-R4-0ubuntu5.5_amd64.deb) ...
Unpacking replacement isc-dhcp-common ...
Preparing to replace sudo 1.8.3p1-1ubuntu3 (using .../sudo_1.8.3p1-1ubuntu3.4_amd64.deb) ...
Unpacking replacement sudo ...
Preparing to replace apt-transport-https 0.8.16~exp12ubuntu10 (using .../apt-transport-https_0.8.16~exp12ubuntu10.10_amd64.deb) ...
Unpacking replacement apt-transport-https ...
Preparing to replace dnsutils 1:9.8.1.dfsg.P1-4 (using .../dnsutils_1%3a9.8.1.dfsg.P1-4ubuntu0.6_amd64.deb) ...
Unpacking replacement dnsutils ...
Preparing to replace bind9-host 1:9.8.1.dfsg.P1-4 (using .../bind9-host_1%3a9.8.1.dfsg.P1-4ubuntu0.6_amd64.deb) ...
Unpacking replacement bind9-host ...
Preparing to replace libisc83 1:9.8.1.dfsg.P1-4 (using .../libisc83_1%3a9.8.1.dfsg.P1-4ubuntu0.6_amd64.deb) ...
Unpacking replacement libisc83 ...
Preparing to replace libdns81 1:9.8.1.dfsg.P1-4 (using .../libdns81_1%3a9.8.1.dfsg.P1-4ubuntu0.6_amd64.deb) ...
Unpacking replacement libdns81 ...
Preparing to replace libisccc80 1:9.8.1.dfsg.P1-4 (using .../libisccc80_1%3a9.8.1.dfsg.P1-4ubuntu0.6_amd64.deb) ...
Unpacking replacement libisccc80 ...
Preparing to replace libisccfg82 1:9.8.1.dfsg.P1-4 (using .../libisccfg82_1%3a9.8.1.dfsg.P1-4ubuntu0.6_amd64.deb) ...
Unpacking replacement libisccfg82 ...
Preparing to replace liblwres80 1:9.8.1.dfsg.P1-4 (using .../liblwres80_1%3a9.8.1.dfsg.P1-4ubuntu0.6_amd64.deb) ...
Unpacking replacement liblwres80 ...
Preparing to replace libbind9-80 1:9.8.1.dfsg.P1-4 (using .../libbind9-80_1%3a9.8.1.dfsg.P1-4ubuntu0.6_amd64.deb) ...
Unpacking replacement libbind9-80 ...
Preparing to replace dbus 1.4.18-1ubuntu1 (using .../dbus_1.4.18-1ubuntu1.4_amd64.deb) ...
Unpacking replacement dbus ...
Preparing to replace krb5-locales 1.10+dfsg~beta1-2 (using .../krb5-locales_1.10+dfsg~beta1-2ubuntu0.3_all.deb) ...
Unpacking replacement krb5-locales ...
Preparing to replace openssl 1.0.1-4ubuntu3 (using .../openssl_1.0.1-4ubuntu5.8_amd64.deb) ...
Unpacking replacement openssl ...
Preparing to replace accountsservice 0.6.15-2ubuntu9 (using .../accountsservice_0.6.15-2ubuntu9.1_amd64.deb) ...
Unpacking replacement accountsservice ...
Preparing to replace libaccountsservice0 0.6.15-2ubuntu9 (using .../libaccountsservice0_0.6.15-2ubuntu9.1_amd64.deb) ...
Unpacking replacement libaccountsservice0 ...
Preparing to replace apache2 2.2.22-1ubuntu1 (using .../apache2_2.2.22-1ubuntu1.3_amd64.deb) ...
Unpacking replacement apache2 ...
Preparing to replace apache2-mpm-prefork 2.2.22-1ubuntu1 (using .../apache2-mpm-prefork_2.2.22-1ubuntu1.3_amd64.deb) ...
 * Stopping web server apache2
[Tue Jul 02 06:34:49 2013] [warn] NameVirtualHost *:80 has no VirtualHosts
 ... waiting    ...done.
Unpacking replacement apache2-mpm-prefork ...
Preparing to replace apache2.2-common 2.2.22-1ubuntu1 (using .../apache2.2-common_2.2.22-1ubuntu1.3_amd64.deb) ...
Unpacking replacement apache2.2-common ...
Preparing to replace apache2.2-bin 2.2.22-1ubuntu1 (using .../apache2.2-bin_2.2.22-1ubuntu1.3_amd64.deb) ...
Unpacking replacement apache2.2-bin ...
Preparing to replace apache2-utils 2.2.22-1ubuntu1 (using .../apache2-utils_2.2.22-1ubuntu1.3_amd64.deb) ...
Unpacking replacement apache2-utils ...
Preparing to replace python-problem-report 2.0.1-0ubuntu5 (using .../python-problem-report_2.0.1-0ubuntu15.1_all.deb) ...
Unpacking replacement python-problem-report ...
Preparing to replace python-apport 2.0.1-0ubuntu5 (using .../python-apport_2.0.1-0ubuntu15.1_all.deb) ...
Unpacking replacement python-apport ...
Preparing to replace apport 2.0.1-0ubuntu5 (using .../apport_2.0.1-0ubuntu15.1_all.deb) ...
apport stop/waiting
Unpacking replacement apport ...
Preparing to replace binutils 2.22-6ubuntu1 (using .../binutils_2.22-6ubuntu1.1_amd64.deb) ...
Unpacking replacement binutils ...
Preparing to replace libgc1c2 1:7.1-8build1 (using .../libgc1c2_1%3a7.1-8ubuntu0.12.04.1_amd64.deb) ...
Unpacking replacement libgc1c2 ...
Preparing to replace ruby1.8-dev 1.8.7.352-2ubuntu1 (using .../ruby1.8-dev_1.8.7.352-2ubuntu1.2_amd64.deb) ...
Unpacking replacement ruby1.8-dev ...
Preparing to replace ruby1.8 1.8.7.352-2ubuntu1 (using .../ruby1.8_1.8.7.352-2ubuntu1.2_amd64.deb) ...
Unpacking replacement ruby1.8 ...
Preparing to replace libruby1.8 1.8.7.352-2ubuntu1 (using .../libruby1.8_1.8.7.352-2ubuntu1.2_amd64.deb) ...
Unpacking replacement libruby1.8 ...
Preparing to replace linux-firmware 1.79 (using .../linux-firmware_1.79.4_all.deb) ...
Unpacking replacement linux-firmware ...
Selecting previously unselected package linux-headers-3.2.0-48.
Unpacking linux-headers-3.2.0-48 (from .../linux-headers-3.2.0-48_3.2.0-48.74_all.deb) ...
Selecting previously unselected package linux-headers-3.2.0-48-virtual.
Unpacking linux-headers-3.2.0-48-virtual (from .../linux-headers-3.2.0-48-virtual_3.2.0-48.74_amd64.deb) ...
Selecting previously unselected package linux-headers-virtual.
Unpacking linux-headers-virtual (from .../linux-headers-virtual_3.2.0.48.58_amd64.deb) ...
Preparing to replace linux-virtual 3.2.0.23.25 (using .../linux-virtual_3.2.0.48.58_amd64.deb) ...
Unpacking replacement linux-virtual ...
Preparing to replace linux-image-virtual 3.2.0.23.25 (using .../linux-image-virtual_3.2.0.48.58_amd64.deb) ...
Unpacking replacement linux-image-virtual ...
Preparing to replace python-crypto 2.4.1-1 (using .../python-crypto_2.4.1-1ubuntu0.1_amd64.deb) ...
Unpacking replacement python-crypto ...
Preparing to replace python-keyring 0.7.1-1fakesync1 (using .../python-keyring_0.9.2-0ubuntu0.12.04.2_all.deb) ...
Unpacking replacement python-keyring ...
Preparing to replace python-software-properties 0.82.7 (using .../python-software-properties_0.82.7.3_all.deb) ...
Unpacking replacement python-software-properties ...
Preparing to replace ri1.8 1.8.7.352-2ubuntu1 (using .../ri1.8_1.8.7.352-2ubuntu1.2_all.deb) ...
Unpacking replacement ri1.8 ...
Preparing to replace update-manager-core 1:0.156.14 (using .../update-manager-core_1%3a0.156.14.5_amd64.deb) ...
Unpacking replacement update-manager-core ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
Processing triggers for ufw ...
Setting up libc-dev-bin (2.15-0ubuntu10.2) ...
Setting up linux-libc-dev (3.2.0-48.74) ...
Setting up libc6-dev (2.15-0ubuntu10.2) ...
Setting up libapt-inst1.4 (0.8.16~exp12ubuntu10.10) ...
Setting up libdbus-1-3 (1.4.18-1ubuntu1.4) ...
Setting up libdrm2 (2.4.39-0ubuntu0.2) ...
Setting up libdrm-intel1 (2.4.39-0ubuntu0.2) ...
Setting up libdrm-nouveau1a (2.4.39-0ubuntu0.2) ...
Setting up libdrm-radeon1 (2.4.39-0ubuntu0.2) ...
Setting up apparmor (2.7.102-0ubuntu3.7) ...
Installing new version of config file /etc/apparmor.d/abstractions/ubuntu-email ...
Installing new version of config file /etc/apparmor.d/abstractions/ubuntu-browsers.d/multimedia ...
Installing new version of config file /etc/apparmor.d/abstractions/nameservice ...
 * Starting AppArmor profiles
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
   ...done.
 * Reloading AppArmor profiles
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
   ...done.
Setting up libgnutls26 (2.12.14-5ubuntu3.4) ...
Setting up libgnutlsxx27 (2.12.14-5ubuntu3.4) ...
Setting up libgnutls-openssl27 (2.12.14-5ubuntu3.4) ...
Setting up libgnutls-dev (2.12.14-5ubuntu3.4) ...
Setting up libcurl3-gnutls (7.22.0-3ubuntu4.1) ...
Setting up libdbus-glib-1-2 (0.98-1ubuntu1.1) ...
Setting up libx11-data (2:1.4.99.1-0ubuntu2.1) ...
Setting up libxcb1 (1.8.1-1ubuntu0.2) ...
Setting up libx11-6 (2:1.4.99.1-0ubuntu2.1) ...
Setting up libxext6 (2:1.3.0-3ubuntu0.1) ...
Setting up libxml2 (2.7.8.dfsg-5.1ubuntu4.4) ...
Setting up libexpat1 (2.0.1-7.2ubuntu1.1) ...
Setting up libfreetype6 (2.4.8-1ubuntu2.1) ...
Setting up linux-image-3.2.0-48-virtual (3.2.0-48.74) ...
Running depmod.
update-initramfs: deferring update (hook will be called later)
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 3.2.0-48-virtual /boot/vmlinuz-3.2.0-48-virtual
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 3.2.0-48-virtual /boot/vmlinuz-3.2.0-48-virtual
update-initramfs: Generating /boot/initrd.img-3.2.0-48-virtual
run-parts: executing /etc/kernel/postinst.d/update-notifier 3.2.0-48-virtual /boot/vmlinuz-3.2.0-48-virtual
run-parts: executing /etc/kernel/postinst.d/x-grub-legacy-ec2 3.2.0-48-virtual /boot/vmlinuz-3.2.0-48-virtual
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-3.2.0-23-virtual
Found kernel: /boot/memtest86+.bin
Found kernel: /boot/vmlinuz-3.2.0-48-virtual
Found kernel: /boot/vmlinuz-3.2.0-23-virtual
Found kernel: /boot/memtest86+.bin
Replacing config file /run/grub/menu.lst with new version
Updating /boot/grub/menu.lst ... done

run-parts: executing /etc/kernel/postinst.d/zz-update-grub 3.2.0-48-virtual /boot/vmlinuz-3.2.0-48-virtual
Generating grub.cfg ...
/usr/sbin/grub-probe: warn: disk does not exist, so falling back to partition device /dev/xvda1.
Found linux image: /boot/vmlinuz-3.2.0-48-virtual
Found initrd image: /boot/initrd.img-3.2.0-48-virtual
/usr/sbin/grub-probe: warn: disk does not exist, so falling back to partition device /dev/xvda1.
/usr/sbin/grub-probe: warn: disk does not exist, so falling back to partition device /dev/xvda1.
/usr/sbin/grub-probe: warn: disk does not exist, so falling back to partition device /dev/xvda1.
/usr/sbin/grub-probe: warn: disk does not exist, so falling back to partition device /dev/xvda1.
Found linux image: /boot/vmlinuz-3.2.0-23-virtual
Found initrd image: /boot/initrd.img-3.2.0-23-virtual
/usr/sbin/grub-probe: warn: disk does not exist, so falling back to partition device /dev/xvda1.
/usr/sbin/grub-probe: warn: disk does not exist, so falling back to partition device /dev/xvda1.
/usr/sbin/grub-probe: warn: disk does not exist, so falling back to partition device /dev/xvda1.
/usr/sbin/grub-probe: warn: disk does not exist, so falling back to partition device /dev/xvda1.
/usr/sbin/grub-probe: warn: disk does not exist, so falling back to partition device /dev/xvda1.
Found memtest86+ image: /boot/memtest86+.bin
done
Setting up apt-utils (0.8.16~exp12ubuntu10.10) ...
Setting up isc-dhcp-common (4.1.ESV-R4-0ubuntu5.5) ...
Setting up isc-dhcp-client (4.1.ESV-R4-0ubuntu5.5) ...
Setting up sudo (1.8.3p1-1ubuntu3.4) ...
Installing new version of config file /etc/pam.d/sudo ...
Setting up apt-transport-https (0.8.16~exp12ubuntu10.10) ...
Setting up libisc83 (1:9.8.1.dfsg.P1-4ubuntu0.6) ...
Setting up libdns81 (1:9.8.1.dfsg.P1-4ubuntu0.6) ...
Setting up libisccc80 (1:9.8.1.dfsg.P1-4ubuntu0.6) ...
Setting up libisccfg82 (1:9.8.1.dfsg.P1-4ubuntu0.6) ...
Setting up libbind9-80 (1:9.8.1.dfsg.P1-4ubuntu0.6) ...
Setting up liblwres80 (1:9.8.1.dfsg.P1-4ubuntu0.6) ...
Setting up bind9-host (1:9.8.1.dfsg.P1-4ubuntu0.6) ...
Setting up dnsutils (1:9.8.1.dfsg.P1-4ubuntu0.6) ...
Setting up dbus (1.4.18-1ubuntu1.4) ...
Setting up krb5-locales (1.10+dfsg~beta1-2ubuntu0.3) ...
Setting up openssl (1.0.1-4ubuntu5.8) ...
Setting up libaccountsservice0 (0.6.15-2ubuntu9.1) ...
Setting up accountsservice (0.6.15-2ubuntu9.1) ...
Setting up apache2.2-bin (2.2.22-1ubuntu1.3) ...
Setting up apache2-utils (2.2.22-1ubuntu1.3) ...
Setting up python-problem-report (2.0.1-0ubuntu15.1) ...
Setting up python-apport (2.0.1-0ubuntu15.1) ...
Setting up apport (2.0.1-0ubuntu15.1) ...
apport start/running
Setting up binutils (2.22-6ubuntu1.1) ...
Setting up libgc1c2 (1:7.1-8ubuntu0.12.04.1) ...
Setting up libruby1.8 (1.8.7.352-2ubuntu1.2) ...
Setting up ruby1.8-dev (1.8.7.352-2ubuntu1.2) ...
Setting up ruby1.8 (1.8.7.352-2ubuntu1.2) ...
Setting up linux-firmware (1.79.4) ...
Setting up linux-headers-3.2.0-48 (3.2.0-48.74) ...
Setting up linux-headers-3.2.0-48-virtual (3.2.0-48.74) ...
Setting up linux-headers-virtual (3.2.0.48.58) ...
Setting up linux-image-virtual (3.2.0.48.58) ...
Setting up linux-virtual (3.2.0.48.58) ...
Setting up python-crypto (2.4.1-1ubuntu0.1) ...
Setting up python-keyring (0.9.2-0ubuntu0.12.04.2) ...
Setting up python-software-properties (0.82.7.3) ...
Setting up ri1.8 (1.8.7.352-2ubuntu1.2) ...
Setting up update-manager-core (1:0.156.14.5) ...
Setting up perl-modules (5.14.2-6ubuntu2.3) ...
Setting up perl (5.14.2-6ubuntu2.3) ...
Setting up apache2.2-common (2.2.22-1ubuntu1.3) ...
Setting up apache2-mpm-prefork (2.2.22-1ubuntu1.3) ...
Setting up apache2 (2.2.22-1ubuntu1.3) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Robbie
  • 91
  • 5
  • It probably failed to start when the package install tried to start it for some reason - do you have the Apache error log from when that happened? – Shane Madden Jul 02 '13 at 15:44
  • @ShaneMadden, thanks, So is it supposed to restart the service automatically then? The only log entered into `/var/log/apache2/error.log` at the time of the error was `[notice] caught SIGTERM, shutting down` on both instances – Robbie Jul 02 '13 at 15:50
  • Yeah, it should start it back up if it's set to start. Looks like you use `apachectl` to start and stop normally, but just to make sure - does it work when you do `service apache2 start`, and does it start the service properly at boot? – Shane Madden Jul 02 '13 at 16:59
  • @ShaneMadden Yeah, `service apache2 restart` works as well - i use `apachectl` out of habit. Also, it does normally start up correctly after boot. In general i don't have issues with starting/stopping/restarting apache... normally – Robbie Jul 02 '13 at 17:07
  • Strange. I wonder if something was just messed up with this particular upgrade package somehow? It should definitely be trying to start the service back up after the upgrade. Unfortunately, it's hard to tell what went wrong in this case because the stdout from the upgrade is lost forever. – Shane Madden Jul 02 '13 at 17:10
  • Think i've just found the stdout, i've edited the post to include it. The Vhost warning needs fixing, but shouldn't be causing this issue i don't think? Anything else in there look suspect to anyone? – Robbie Jul 02 '13 at 17:25
  • Oh, nice of it to keep the output! I'll dig a little deeper into what the package scripts are doing when I have a chance later on today. – Shane Madden Jul 02 '13 at 17:40

2 Answers2

2

I ran into the exact same problem today. It's late answer but might be helpful to someone.
My OS: ubuntu 16.04.7

Short answer:
DO NOT start apache2 processes by apachectl start.
Instead, always use systemctl start apache2.


When you start apache2 by apachectl systemd cannot realize the processes are running. (systemctl status apache2 shows you inactive(dead))

One day, unattended-upgrades applies some library updates, then apache2 is going to be restarted via systemd. This behavior is defined in /var/lib/dpkg/info/xxx.postinst (xxx is the library updated). In my case apache2_invoke enmod php7.0 is called and finally it invokes invoke-rc.d apache2 restart which is equivalent to systemctl stop apache2 && systemctl start apache2.

Let's take a look at the systemd service unit of apache2.

/lib/systemd/system/apache2.service.d/apache2-systemd.conf

[Service]
Type=forking
RemainAfterExit=no

The definition has no ExecStart of ExecStop line because it is generated by systemd-sysv-generator, so systemctl cmd is a facade of /etc/init.d/apache2 in this case.
The important line is RemainAfterExit=no. This means "After the start script (init.d/apache2 start) exits, there must not be the daemon either, so that ExecStop will be called to clean up processes."

Here is a whole apache-disappearing scenario.

  1. unattended-upgrades invokes systemctl restart apache2. (or maybe just systemctl start apahce2, then skip to 3)
  2. ExecStop is called and just ignored because the service status is inactive.
  3. ExecStart is called and exits immediately because there are already apache2 processes which started by apachectl.
  4. ExecStop is called again because ExecStart is over, this time the script is executed because systemd now controls apache2.
  5. ExecStop (/etc/init.d/apache2 stop) kills all apache2 processes even though they are started by apachectl.
0

I have been checking the postinst scripts (in /var/lib/dpkg/info/) of the packages you upgraded that are also related to apache, and no one of them launches apache, the only file where i have seen it is in :

apache2-mpm-worker.postinst

Which is not in the packages being upgraded. This looks like the package maintainer didn't take into account that, following the dependencies, the apache process stopped by apache2-mpm-prefork should be started later.

blu3g3
  • 86
  • 4
  • thanks, in `/var/lib/dpkg/info/apache2-mpm-prefork.postinst` i see a call to `invoke-rc.d apache2 start ||true` near the end. So it would seem that this should work...confused – Robbie Jul 16 '13 at 18:10