MySQL Wont Start with Datadir Change

Question

I'm trying to change the datadir for mysql. When I try to restart the service it hangs for 10 minutes, then fails to restart. Here are the steps I took to change the dir:

sudo service mysqld stop
sudo cp -R -p /var/lib/mysql /srv/msqyl2
ls -lha /srv/mysql2

drwxr-xr-x. 4 mysql mysql 4.0K Aug 24 17:51 msqyl2

sudo vim /etc/my.cnf
Comment out default directory, `#datadir=/var/lib/mysql`
Add a new line with same directive but new location, `datadir=/srv/mysql2`
sudo service mysql start

This gives me:

Job for mysqld.service failed because a timeout was exceeded. See "systemctl status mysqld.service" and "journalctl -xe" for details.

after about 10 minutes of it processing the start. If I reverse the commenting so the directory is the default directory, then the restart works in < 5 seconds.

The journalctl -xe brings back:

Aug 24 18:35:17 uroot-ARTMO-myserver-1 systemd[1]: mysqld.service start-post operation timed out. Stopping.
Aug 24 18:35:17 uroot-ARTMO-myserver-1 systemd[1]: Failed to start MySQL Community Server.
-- Subject: Unit mysqld.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mysqld.service has failed.
-- 
-- The result is failed.
Aug 24 18:35:17 uroot-ARTMO-myserver-1 systemd[1]: Unit mysqld.service entered failed state.
Aug 24 18:35:18 uroot-ARTMO-myserver-1 systemd[1]: mysqld.service failed.
Aug 24 18:35:18 uroot-ARTMO-myserver-1 polkitd[1309]: Unregistered Authentication Agent for unix-process:22433:121785532 (system bus name :1.833, object path /org/freedesktop/PolicyKi
Aug 24 18:35:18 uroot-ARTMO-myserver-1 systemd[1]: mysqld.service holdoff time over, scheduling restart.
Aug 24 18:35:18 uroot-ARTMO-myserver-1 systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysqld.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mysqld.service has begun starting up.
Aug 24 18:35:18 uroot-ARTMO-myserver-1 mysqld_safe[24090]: 160824 18:35:18 mysqld_safe Logging to '/var/log/mysqld.log'.
Aug 24 18:35:18 uroot-ARTMO-myserver-1 mysqld_safe[24090]: 160824 18:35:18 mysqld_safe Starting mysqld daemon with databases from /srv/mysql2
Aug 24 18:35:18 uroot-ARTMO-myserver-1 mysqld_safe[24090]: 160824 18:35:18 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

The systemctl status mysqld.service brings back:

● mysqld.service - MySQL Community Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: activating (start-post) since Wed 2016-08-24 18:45:18 CEST; 8min ago
  Process: 26133 ExecStart=/usr/bin/mysqld_safe (code=exited, status=0/SUCCESS)
  Process: 26120 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 26133 (code=exited, status=0/SUCCESS);         : 26134 (mysql-systemd-s)
   CGroup: /system.slice/mysqld.service
           └─control
             ├─26134 /bin/bash /usr/bin/mysql-systemd-start post
             └─27942 sleep 1

Aug 24 18:45:18 uroot-ARTMO-myserver-1 systemd[1]: Starting MySQL Community Server...
Aug 24 18:45:18 uroot-ARTMO-myserver-1 mysqld_safe[26133]: 160824 18:45:18 mysqld_safe Logging to '/var/log/mysqld.log'.
Aug 24 18:45:18 uroot-ARTMO-myserver-1 mysqld_safe[26133]: 160824 18:45:18 mysqld_safe Starting mysqld daemon with databases from /srv/mysql2

The permissions for the default dir /var/lib/mysql is:

drwx------. 2 mysql mysql 4.0K Aug 18 21:32 mysql

The mysqld.log has:

160824 18:35:18 mysqld_safe Starting mysqld daemon with databases from /srv/mysql2
2016-08-24 18:35:18 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-08-24 18:35:18 0 [Note] /usr/sbin/mysqld (mysqld 5.6.32) starting as process 24253 ...
2016-08-24 18:35:18 24253 [Warning] Can't create test file /srv/mysql2/uroot-ARTMO-myserver-1.lower-test
2016-08-24 18:35:18 24253 [Warning] Can't create test file /srv/mysql2/uroot-ARTMO-myserver-1.lower-test
2016-08-24 18:35:18 24253 [Warning] Buffered warning: Changed limits: max_open_files: 1024 (requested 5000)

2016-08-24 18:35:18 24253 [Warning] Buffered warning: Changed limits: table_open_cache: 431 (requested 2000)

2016-08-24 18:35:18 24253 [Note] Plugin 'FEDERATED' is disabled.
/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13 - Permission denied)
2016-08-24 18:35:18 24253 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2016-08-24 18:35:18 24253 [Note] InnoDB: Using atomics to ref count buffer pool pages
2016-08-24 18:35:18 24253 [Note] InnoDB: The InnoDB memory heap is disabled
2016-08-24 18:35:18 24253 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-08-24 18:35:18 24253 [Note] InnoDB: Memory barrier is not used
2016-08-24 18:35:18 24253 [Note] InnoDB: Compressed tables use zlib 1.2.3
2016-08-24 18:35:18 24253 [Note] InnoDB: Using Linux native AIO
2016-08-24 18:35:18 24253 [Note] InnoDB: Using CPU crc32 instructions
2016-08-24 18:35:18 24253 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-08-24 18:35:18 24253 [Note] InnoDB: Completed initialization of buffer pool
2016-08-24 18:35:18 7f32872e8740  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
2016-08-24 18:35:18 24253 [ERROR] InnoDB: os_file_get_status() failed on './ibdata1'. Can't determine file permissions
2016-08-24 18:35:18 24253 [ERROR] InnoDB: The system tablespace must be writable!
2016-08-24 18:35:18 24253 [ERROR] Plugin 'InnoDB' init function returned error.
2016-08-24 18:35:18 24253 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2016-08-24 18:35:18 24253 [ERROR] Unknown/unsupported storage engine: InnoDB
2016-08-24 18:35:18 24253 [ERROR] Aborting

2016-08-24 18:35:18 24253 [Note] Binlog end
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'partition'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_METRICS'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_CMPMEM'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_CMP'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_LOCKS'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'INNODB_TRX'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'BLACKHOLE'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'ARCHIVE'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'MRG_MYISAM'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'MyISAM'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'MEMORY'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'CSV'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'sha256_password'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'mysql_old_password'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'mysql_native_password'
2016-08-24 18:35:18 24253 [Note] Shutting down plugin 'binlog'
2016-08-24 18:35:18 24253 [Note] /usr/sbin/mysqld: Shutdown complete

160824 18:35:18 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

I see no `/var/log/audit.log` and `semanage` isn't configured on the server, should it be? — chris85, Aug 24 '16 at 20:25
It should be /var/log/audit/audit.log and semanage is in policycoreutils-python which may not be installed. — user9517, Aug 24 '16 at 20:32
Thanks, with `yum install policycoreutils-python` I was able to use that other answer. — chris85, Aug 24 '16 at 20:36

score 3 · Answer 1 · answered Aug 24 '16 at 18:30

3

Often time this is because of something like apparmor. Check in /etc/ if there are apparmor, selinux, or whatever profiles, restricting mysql to /var/lib/mysql.

answered Aug 24 '16 at 18:30

Halfgaar

8,084
6
45
86

Yes, I have `selinux`. I found `/etc/selinux/targeted/contexts/files/file_contexts` Is this file a list of directories mysql can execute from? – chris85 Aug 24 '16 at 20:08
@chris85 I don't have experience with selinux, but in general, yes, you have to whitelist paths it can access. – Halfgaar Aug 24 '16 at 21:01

score 2 · Answer 2 · answered Aug 24 '16 at 18:45

2

If the commands you posted in your question are correct, then you have a typo.

sudo cp -R -p /var/lib/mysql /srv/msqyl2
ls -lha /srv/mysql2
drwxr-xr-x. 4 mysql mysql 4.0K Aug 24 17:51 msqyl2
Add a new line with same directive but new location, `datadir=/srv/mysql2`

mysql2 is not msqyl2

answered Aug 24 '16 at 18:45

Zoredache

130,897
41
276
420

Yup, typo was in my code, modified to `sudo cp -R -p /var/lib/mysql /srv/mysql3` and changed the `datadir` but still the same affect. – chris85 Aug 24 '16 at 20:04

MySQL Wont Start with Datadir Change

2 Answers2

Linked