MySQL 5.6 on CentOS 6 silently fails to start

Question

I'm working on a fresh CentOS 6.5 install and trying to get a MySQL 5.6 server daemon running. The server is a fresh droplet on DigitalOcean's infrastructure. 512 MB size. I used the MySQL version from the MySQL Yum repository. When I try to start it, it mysteriously crashes without any indication of what went wrong. Here are some details:

First, my /etc/my.cnf:

[mysqld]
innodb_buffer_pool_size = 5M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES 
explicit_defaults_for_timestamp = TRUE

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

When I run service mysqld start, I get this:

[root@myserver ~]# service mysqld start
MySQL Daemon failed to start.
Starting mysqld:                                           [FAILED]

And here's the complete contents of /var/log/mysqld.log after entering that command:

140104 22:53:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2014-01-04 22:53:42 8177 [Note] Plugin 'FEDERATED' is disabled.
2014-01-04 22:53:42 8177 [Note] InnoDB: The InnoDB memory heap is disabled
2014-01-04 22:53:42 8177 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-01-04 22:53:42 8177 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-01-04 22:53:42 8177 [Note] InnoDB: Using Linux native AIO
2014-01-04 22:53:42 8177 [Note] InnoDB: Not using CPU crc32 instructions
2014-01-04 22:53:42 8177 [Note] InnoDB: Initializing buffer pool, size = 5.0M
2014-01-04 22:53:42 8177 [Note] InnoDB: Completed initialization of buffer pool
2014-01-04 22:53:42 8177 [Note] InnoDB: Highest supported file format is Barracuda.
140104 22:53:43 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

No errors, no warnings, no nothing. And when I run service mysqld status, it says mysqld dead but subsys locked.

Any ideas?

Update: I went into /etc/init.d/mysqld and added a line to echo the exact startup command it's using to the console. I then ran that command (but without redirecting any output to /dev/null or backgrounding the process). Here's what turns up:

[root@myserver ~]# /usr/bin/mysqld_safe --datadir="/var/lib/mysql" --socket="/var/lib/mysql/mysql.sock" --pid-file="/var/run/mysqld/mysqld.pid" --basedir=/usr --user=mysql
140105 08:04:35 mysqld_safe Logging to '/var/log/mysqld.log'.
140105 08:04:35 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/bin/mysqld_safe: line 166: 10966 Killed                  nohup /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock < /dev/null >> /var/log/mysqld.log 2>&1
140105 08:04:36 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

It looks like some external service or watchdog is killing MySQL as soon as it tries to start up. But what could be doing this? SELinux doesn't appear to be installed. I don't know what else...?

Answer 1

It was definitely a memory issue. This became evident once I looked in /var/log/messages. I was trying to run recent versions of Nginx, PHP-FPM, and MySQL, and the default configurations for all 3 of these together were too much for my little droplet that had only 512 MB of memory and no swap space.

I tweaked my /etc/my.cnf and my PHP-FPM config, and everything began working fine. Here's my new my.cnf for reference purposes:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
explicit_defaults_for_timestamp = TRUE
key_buffer=16K
table_open_cache=4
query_cache_limit=256K
query_cache_size=4M
max_allowed_packet=1M
sort_buffer_size=64K
read_buffer_size=256K
thread_stack=64K
innodb_buffer_pool_size = 56M
innodb_flush_neighbors=0
innodb_flush_log_at_trx_commit=2
character-set-server=utf8
collation-server=utf8_general_ci

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

The various parameters I added were cobbled together from various blogs and tutorial sites focused on low-memory mysqld deployments, such as here and here.

Answer 2

The first thing I would check here is SELinux. First off set it into Permissive mode

setenforce 0

Then try starting mysql. If it starts then we know for sure that SELinux is the culprit. If this is the case then /var/log/audit/audit.log will contain AVC denied messages relating to MySQL that will help you diagnose the problem further.

Don't forget to renable SELinux enforcing mode.

setenforce 1