2

I'm having an issue where MySQL 5.1.54 is restarting every 30 minutes on Ubuntu 11.04. When this occurs, the following appears in the MySQL log:

111030 12:01:52 [Note] /usr/sbin/mysqld: Normal shutdown

111030 12:01:52 [Note] Event Scheduler: Purging the queue. 0 events
111030 12:01:52  InnoDB: Starting shutdown...
111030 12:01:54  InnoDB: Shutdown completed; log sequence number 0 875122
111030 12:01:54 [Note] /usr/sbin/mysqld: Shutdown complete

111030 12:01:55 [Note] Plugin 'FEDERATED' is disabled.
111030 12:01:55  InnoDB: Initializing buffer pool, size = 256.0M
111030 12:01:55  InnoDB: Completed initialization of buffer pool
111030 12:01:55  InnoDB: Started; log sequence number 0 875122
111030 12:01:55 [Note] Event Scheduler: Loaded 0 events
111030 12:01:55 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.54-1ubuntu4-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)

This occurs like clockwork every 30 minutes, so it's obviously some service restarting it.

I have checked the crontab of every user on the system (including system users), and none of them have a crontab setup, as you can see in the output below:

# awk -F: '{print $1}' /etc/passwd | xargs -n 1 -i crontab -u {} -l  
no crontab for root
no crontab for daemon
no crontab for bin
no crontab for sys
no crontab for sync
no crontab for games
no crontab for man
no crontab for lp
no crontab for mail
no crontab for news
no crontab for uucp
no crontab for proxy
no crontab for www-data
no crontab for backup
no crontab for list
no crontab for irc
no crontab for gnats
no crontab for nobody
no crontab for libuuid
no crontab for syslog
no crontab for sshd
no crontab for landscape
no crontab for ubuntu
no crontab for statd
no crontab for myproxy
no crontab for condor
no crontab for messagebus
no crontab for avahi
no crontab for joe
no crontab for smmta
no crontab for smmsp
no crontab for postfix
no crontab for deploy
no crontab for mysql
no crontab for redis

My dmesg contains the following each time it is restarted. I'm not an apparmor expert, but I believe this is a normal message obtained each time the MySQL service starts:

[1165328.780405] type=1400 audit(1319976114.984:74): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=31985 comm="apparmor_parser"

Also, here are the contents of the MySQL upstart configuration in /etc/init/mysql.conf:

# MySQL Service

description     "MySQL Server"
author          "Mario Limonciello <superm1@ubuntu.com>"

start on (net-device-up
          and local-filesystems
      and runlevel [2345])
stop on runlevel [016]

respawn

env HOME=/etc/mysql
umask 007

# The default of 5 seconds is too low for mysql which needs to flush buffers
kill timeout 300

pre-start script
    #Sanity checks
    [ -r $HOME/my.cnf ]
    [ -d /var/run/mysqld ] || install -m 755 -o mysql -g root -d /var/run/mysqld
    /lib/init/apparmor-profile-load usr.sbin.mysqld
    LC_ALL=C BLOCKSIZE= df --portability /var/lib/mysql/. | tail -n 1 | awk '{ exit ($4<4096) }'
end script

exec /usr/sbin/mysqld

post-start script
   for i in `seq 1 30` ; do
        /usr/bin/mysqladmin --defaults-file="${HOME}"/debian.cnf ping && {
            exec "${HOME}"/debian-start
            # should not reach this line
            exit 2
        }
        sleep 1
    done
    exit 1
end script

Any idea what might be causing this? It doesn't cause any problems, other than Monit alerts stating that "PID changed Service mysqld" (I have Monit monitoring mysqld -- but it reports no errors with the mysqld process, other than the fact that every 30 minutes, it has its PID changed since MySQL is restarted).

Thanks in advance.

Michael Durrant
  • 93,410
  • 97
  • 333
  • 497
user1020643
  • 21
  • 2
  • 3

5 Answers5

1

are you using chef or puppet, which might be doing something that triggers the reboot?

Tom
  • 3,324
  • 1
  • 31
  • 42
0

Seems converting the /etc/init.d/mysql start up script to sysV style rather than as an upstart script seemed to correct the problem for me.

Frank
  • 1
  • 1
0

I had this same problem upgrading to Ubuntu 14.04. I found this question because it mentions the AppArmor log message, so thanks! I may not have realised MySQL was restarting otherwise.

Upon investigating /var/log/daemon.log, I found the output of /etc/mysql/debian-start appearing repeatedly. The relevant part was this:

May 18 06:48:18 tom /etc/mysql/debian-start[15525]: Upgrading MySQL tables if necessary.
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Looking for 'mysql' as: /usr/bin/mysql
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Error: Server version (5.5.35-1ubuntu1) does not match with the version of
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: the server (5.5.37) with which this program was built/distributed. You can
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: use --skip-version-check to skip this check.
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: FATAL ERROR: Upgrade failed

I read through the /etc/mysql/debian-start script and tried running the upgrade command as in the script, hoping to debug it (MySQL server needs to be running at the time):

/usr/bin/mysql_upgrade --defaults-extra-file=/etc/mysql/debian.cnf

I then found that this worked without complaint and everything simply worked from that point onward. I don't know why it was failing in the first place, but that seemed to fix it. MySQL hasn't restarted itself since.

Stuart Caie
  • 2,803
  • 14
  • 15
0

can you check (and probably post) the definition of your mysql job in upstart? (/etc/init/mysql.conf). OK = try removing the "respawn". It does not work as documented in the upstart documentation. Generally it's used to respawn the process if it's killed by other process, but it seems it does not function as expected. You can see why the apparmour is always loading - because of the pre-start stanza in the script. As upstart is very new, and is still evolving, better is to use the SysV way.

hovanessyan
  • 30,580
  • 6
  • 55
  • 83
  • Tried removing it just now and restarted MySQL. I agree on sticking with the SysV way. However, this is an out-of-the-box upstart script packaged with the standard mysql-server package on Ubuntu 11.04, so one would expect it to work. Nevertheless, I'll know in the next 30 minutes or so whether or not the problem persists. Thanks for your suggestion. – user1020643 Oct 31 '11 at 01:20
  • Nope, it restarted almost exactly 30 minutes after I removed 'respawn' from the upstart script and restarted it manually. :( – user1020643 Oct 31 '11 at 01:48
  • than disable also the apparmour profile. I have also relied on this 'out-of-the-box' upstart scripts, but they do not always behave as expected. If you want to stick with the upstart script experiment with the following - copy the current upstart job definition, and strip it to the bare minimum (runlevels when to run the script + start service command) - than use that script. After you see the result, start to add the other stuff (post and pre start). If this does not help, your problem is caused by another service and not MYSQL itself. – hovanessyan Oct 31 '11 at 06:35
0

You should try to run it without AppArmor: just run /usr/bin/mysqld_safe or /usr/bin/mysqld without using upstart and wait 30 minutes. If mysql does not auto-restart, then disable AppArmor in the /etc/init/mysql.conf file, or configure it differently.

If the problem is still there, read mysql's log. If the log is not enabled by default, you can use the option --log-error=/tmp/mysql.log --log-warnings when launching mysqld.