Duration of starting Mysqld is too long on Lustre file system (InnoDB: Unable to lock ./ibdata1, error: 38)

Question

I can start mysqld and use it normally. But the duration of start is very long (over 3 minuts). When I check log file (/var/log/mysqld.log), I found InnoDB: Unable to lock ./ibdata1, error: 38.

Recently, I moved my mysql data from /var/lib/mysql to /home/user/mysql because files are too large. Then I changed datadir in /etc/my.cnf and /etc/init.d/mysqld respectively and changed datasock in etc/my.cnf. Owner and mod of /home/user/mysql and files in it is correctly set as well.

The /home volume in this installation is on the Lustre file system.

When I found there are 3 innodb engine table in my database, I droped them. But the problem is still here.

Here is the log when starting mysqld (/var/log/mysqld.log).

141027 19:40:03 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
141027 19:40:04 mysqld_safe Starting mysqld daemon with databases from /home/user/mysql
InnoDB: Unable to lock ./ibdata1, error: 38
141027 19:40:04  InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
InnoDB: Unable to lock ./ibdata1, error: 38
141027 19:41:45  InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
141027 19:41:45  InnoDB: Operating system error number 38 in a file operation.
InnoDB: Error number 38 means 'Function not implemented'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not InnoDB: remove old data files which contain your precious data!
141027 19:41:45 [ERROR] Plugin 'InnoDB' init function returned error.
141027 19:41:45 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
141027 19:41:45 [Note] Event Scheduler: Loaded 0 events
141027 19:41:45 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.52'  socket: '/home/user/mysql/mysql.sock'  port: 3306  Source distribution

The error log messages contain some instructions for troubleshooting and correcting the problem. Have you followed those instructions? If so, what happened? — O. Jones, Oct 27 '14 at 12:23
Is your `/home/` filesystem on some different kind of volume (e.g. nfs rather that extfs4)? That *might* account for error 38 -- `function not implemented`. — O. Jones, Oct 27 '14 at 12:24
@OllieJones I did those instructions but the error is still here. /home is on a shared storage and /var/lib/mysql is local. I moved the mysql directory from local to a shared storage .Is this the main reason? — BooksE, Oct 28 '14 at 00:57
@OllieJones I checked the type of filesystem and found my local filesystem is ext4 but the shared storage is lustre. Do you have any idea to fix this problem without dumping all table out? — BooksE, Oct 28 '14 at 07:10

score 1 · Answer 1 · answered Oct 28 '14 at 12:49

You're using a Lustre file system for your /home volume. Unless you enable it specifically on a Lustre volume, that file system doesn't support file locking. But InnoDB needs to lock its files to function correctly. So when InnoDB issues the ioctl(2) calls to lock the file, the file system kicks back "Function not implemented." That's what your error log says.

InnoDB: Error in opening ./ibdata1
141027 19:41:45  InnoDB: Operating system error number 38 in a file operation.
InnoDB: Error number 38 means 'Function not implemented'.

Here's a link to an old listserv on this point. http://lists.lustre.org/pipermail/lustre-discuss/2007-August/003768.html Somewhere in that thread somebody mentions how to enable file locking.

You may want to work with the people administering your file systems to create a special-purpose Lustre volume for your MySQL database, a volume that has locking enabled.

Is this a mission-critical MySQL server? If so, please do a lot of due-diligence before deploying it on this file system.

Thank you for your answer. It is not easy to push administrator to create a special Lustre volume for MySQL database. In fact, I do not need InnoDB Engine and i don't have any table whose engine is InnoDB. Is there some way to disable InnoDB engine and skip the locking of ibdata1? — BooksE, Oct 29 '14 at 01:04
After read url provide by you. I know how to disable InnoDB engine. add skip-innodb in /etc/my.cnf. Thanks again. And I'm new here. When i earn enough reputation, I will vote and accept your answer. — BooksE, Oct 29 '14 at 01:14

Duration of starting Mysqld is too long on Lustre file system (InnoDB: Unable to lock ./ibdata1, error: 38)

1 Answers1