Mysql won't start - ibdata1 corrupt? - operating system error number 13 - permissions issue

Question

Server shutdown from power failure.
Mysql will not start now.
Disk is not full. Syslog is below

Oct 11 15:03:31 joe mysqld_safe[24757]: started
Oct 11 15:03:31 joe mysqld[24760]: 101011 15:03:31  InnoDB: Operating system error number 13 in a file operation.
Oct 11 15:03:31 joe mysqld[24760]: InnoDB: The error means mysqld does not have the access rights to
Oct 11 15:03:31 joe mysqld[24760]: InnoDB: the directory.
Oct 11 15:03:31 joe mysqld[24760]: InnoDB: File name ./ibdata1
Oct 11 15:03:31 joe mysqld[24760]: InnoDB: File operation call: 'create'.
Oct 11 15:03:31 joe mysqld[24760]: InnoDB: Cannot continue operation.

Answer 1

If you are using ubuntu or apparmor you should permit this change in apparmor.

Edit /etc/apparmor.d/usr.sbin.mysqld and change /var/lib/mysql with the new DATADIR.

It should work.

Answer 2

Error:

101130 14:42:51 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
101130 18:07:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
101130 18:07:58  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
InnoDB: File name ./ibdata1
InnoDB: File operation call: 'open'.
InnoDB: Cannot continue operation.

Solution SeLinux SeLinux security:

[root@localhost ~]# service mysqld restart
Deteniendo mysqld:                                         [  OK  ]
Iniciando mysqld:                                          [  FALLÓ  ]
[root@localhost ~]#  restorecon -R /var/lib/mysql/
[root@localhost ~]# service mysqld restart
Deteniendo mysqld:                                         [  OK  ]
Iniciando mysqld:                                          [  OK  ]
[root@localhost ~]#

Answer 3

please check this:

chown -R mysql:mysql /var/lib/mysql

Answer 4

For me, restoring the security context (selinux) did the trick

restorecon -R /var/lib/mysql/

Answer 5

The file is not corrupt. You can find out the source of these errors with 'perror'. i.e.

toaster:~ morgo$ perror 13
OS error code  13:  Permission denied

InnoDB has corruption detection (page checksums) and would happily tell you if that were the problem.

Either the directory permissions have changed, or your my.cnf file has been hosed, and it's trying to recreate data files somewhere else.

Answer 6

In short, (especially on RHEL/CentOS/Fedora) try

getenforce

if it replies with Enforcing you have SELinux up and running. Temporarily deactivate it with setenforce 0 and see if MariaDB starts now! Rather common, especially on RHEL/CentOS/Fedora.

There's more about this further down, as well as in this official article.

In general

There are more things in a UNIX environment that might prevent file access, than just user access rights.

• Security modules like SELinux (see above) or AppArmor (as Dan mentioned) could disallow it
• Access Control Lists (ACL) could be specifically set, for the required files/directories
• Any of the parent folders could be owned by another user, and have no x (="dir access") set for others

Additionally there could be other unexpected factors, like ...

• The mysql datadir being set to a place, where mysql doesn't have permissions (see /etc/my.cnf)
• Mysql could (strangely) be running as a different user, or the file could be simply owned by someone else

Just to mention a view things off the top of my head (feel free to edit/add to this answer btw).

In the case, SELinux is "the problem"

For a permanent solution, you could try to restore the appropriate security context, ...

restorecon -R /var/lib/mysql/

... or just deactivate SELinux (but think about this one a little bit before doing so), by editing the config (typically in /etc/selinux/config) and setting SELINUX=disabled as suggested in following article.

• Here the official help page from mariadb.com: What to do if MariaDB doesn't start
• And here something from redhat.com: MariaDB Changing Database Location

Obviously those are applicable to MySQL just the same way.

Answer 7

I had exactly the same problem on my CentOS box. After moving mysql data directory around I couldn't start the service anymore, even as I had copied the files with the same owner and permissions.

I had a problem with the SELinux security context. If you run your CentOS stock it has good chance to be enabled and won't let do what you want with MySQL. To fix this :

First compare the old dir and new dir using

ls -Z /var/lib/mysql 

and

ls -Z /new/mysql/dir

If you see any difference it's likely to be your problem. To modify this :

chcon -R --type=mysql_db_t /new/mysql/dir

The -R switch is for recursion. If you only need to change one file you can omit it. If your context is different than mine(maybe a different distro), use the one indicated by the output of the first (it should be the 3rd field of the SELinux stuff)

ls -Z /var/lib/mysql 

Answer 8

I had the same problem and fix by below steps

Working directory /var/lib/mysql

Earlier /var/lib/mysql was owned by some unknown user

Changed it to mysql

mysql]# chown -R mysql:mysql *

mysql]# service mariadb start

Redirecting to /bin/systemctl start mariadb.service

Works like a charm

Answer 9

When this popped up for me, I found the answer in the /etc/mysql/my.cnf configuration file. The datadir line did not point to the /var/lib/mysql directory (where the databases are). Once I put this path in, the server restarted no problem.

Answer 10

I had exactly the same problem on my CentOS box. After moving mysql data directory around I couldn't start the service anymore, even as I had copied the files with the same owner and permissions.

I had a problem with the SELinux security context. If you run your CentOS stock it has good chance to be enabled and won't let do what you want with MySQL. To fix this :

First compare the old dir and new dir using

ls -Z /var/lib/mysql 

and

ls -Z /new/mysql/dir

If you see any difference it's likely to be your problem. To modify this :

chcon -R --type=mysql_db_t /new/mysql/dir

The -R switch is for recursion. If you only need to change one file you can omit it.

Answer 11

If you use SEL Linux

Intall semanage

yum whatprovides /usr/sbin/semanage you get policycoreutils-python-2.5-22.el7.x86_64

See mysqld security context

After installation yum install policycoreutils-python you can just look what different security context mysqld has.

semanage fcontext -l | grep mysqld
/etc/mysql(/.*)?                       all files    system_u:object_r:mysqld_etc_t:s0
/etc/my\.cnf\.d(/.*)?                  all files    system_u:object_r:mysqld_etc_t:s0
/var/log/mysql.*                       regular file system_u:object_r:mysqld_log_t:s0
/var/lib/mysql(-files|-keyring)?(/.*)? all files    system_u:object_r:mysqld_db_t:s0
/var/run/mysqld(/.*)?                  all files    system_u:object_r:mysqld_var_run_t:s0
/var/log/mariadb(/.*)?                 all file     system_u:object_r:mysqld_log_t:s0
/var/run/mariadb(/.*)?                 all files    system_u:object_r:mysqld_var_run_t:s0
/usr/sbin/mysqld(-max)?                regular file system_u:object_r:mysqld_exec_t:s0
/var/run/mysqld/mysqlmanager.*         regular file system_u:object_r:mysqlmanagerd_var_run_t:s0
/usr/lib/systemd/system/mysqld.*       regular file system_u:object_r:mysqld_unit_file_t:s0
/usr/lib/systemd/system/mariadb.*      regular file system_u:object_r:mysqld_unit_file_t:s0
/etc/my\.cnf                           regular file system_u:object_r:mysqld_etc_t:s0
/root/\.my\.cnf                        regular file system_u:object_r:mysqld_home_t:s0
/usr/sbin/ndbd                         regular file system_u:object_r:mysqld_exec_t:s0
/usr/libexec/mysqld                    regular file system_u:object_r:mysqld_exec_t:s0
/usr/bin/mysqld_safe                   regular file system_u:object_r:mysqld_safe_exec_t:s0
/usr/bin/mysql_upgrade                 regular file system_u:object_r:mysqld_exec_t:s0
/etc/rc\.d/init\.d/mysqld              regular file system_u:object_r:mysqld_initrc_exec_t:s0
/var/lib/mysql/mysql\.sock             socket       system_u:object_r:mysqld_var_run_t:s0
/usr/libexec/mysqld_safe-scl-helper    regular file system_u:object_r:mysqld_safe_exec_t:s0
/home/[^/]+/\.my\.cnf                  regular file unconfined_u:object_r:mysqld_home_t:s0

Here you see all context for mysqld a short list with explanation

1. mysqld_etc_t - config files
2. mysqld_db_t - data db files
3. mysqld_log_t - log files
4. mysqld_exec_t - execution files

So if you have the wrong security context on your files you get a permission denied (error 13)

Solution

chcon -R -u system_u -t mysqld_db_t  /var/lib/mysql

But check the "normal" permissions, too. I had this problem with centos. You have to systemctl restart mysql for the changes.

Answer 12

In my siuation is Selinux's problem. And the
chcon -R --type=mysql_db_t /new/mysql/dir comes error:
chcon: failed to change context of /new/mysql/dir to root:object_r:mysql_db_t: Invalid argument.
So i use the command：chcon -R root:object_r:mysqld_db_t /new/mysql/dir.

Answer 13

If you have this problem on a Synology NAS you can fix it by following the advice of the Synology support team:

Dear User,

This has been confirmed as a known issue and we will try to fix this issue in further MariaDB release. Sorry for your inconvenience.

Here is the workaround:

• Please try to telnet to your DS with "root" account and password (same as admin's)
• run the command "echo 1 > /var/services/mysql/VERSION"
• open MariaDB package from DSM will trigger update again
• type in the DB password and click Update will fix this issue

More info: Synology forum

Answer 14

I had the same problem. Did alot of research and found out this solution. You need to run this command on ibdata1 sudo shadowprotect -u root | root

I dont know what this does.. but it worked for me.

Good luck.