3

I'm trying to backup using Percona XtraBackup and getting the following error:

[root@ads-dev1 ~]# innobackupex /root/db

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013.  All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

140722 11:52:16  innobackupex: Starting mysql with options:  --unbuffered --
140722 11:52:16  innobackupex: Connected to database with mysql child process (pid=15930)
140722 11:52:22  innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql  Ver 14.12 Distrib 5.0.95, for redhat-linux-gnu (x86_64) using readline 5.1
innobackupex: Using mysql server version Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

innobackupex: Created backup directory /root/db/2014-07-22_11-52-22
140722 11:52:22  innobackupex: Starting mysql with options:  --unbuffered --
140722 11:52:22  innobackupex: Connected to database with mysql child process (pid=15953)
140722 11:52:24  innobackupex: Connection to database server closed

140722 11:52:24  innobackupex: Starting ibbackup with command: xtrabackup_51  --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/root/db/2014-07-22_11-52-22 --tmpdir=/tmp
innobackupex: Waiting for ibbackup (pid=15959) to suspend
innobackupex: Suspend file '/root/db/2014-07-22_11-52-22/xtrabackup_suspended'

xtrabackup_51 version 2.0.8 for MySQL server 5.1.59 unknown-linux-gnu (x86_64) (revision id: 587)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 3
xtrabackup:   innodb_log_file_size = 536870912
xtrabackup: using O_DIRECT
>> log scanned up to (94 3209639289)
140722 11:52:25  InnoDB: Operating system error number 24 in a file operation.
InnoDB: Error number 24 means 'Too many open files'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
InnoDB: Error: could not open single-table tablespace file
InnoDB: ./roei_cc/transport.ibd!
InnoDB: We do not continue the crash recovery, because the table may become
InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
InnoDB: To fix the problem and start mysqld:
InnoDB: 1) If there is a permission problem in the file and mysqld cannot
InnoDB: open the file, you should modify the permissions.
InnoDB: 2) If the table is not needed, or you can restore it from a backup,
InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
InnoDB: crash recovery and ignore that table.
InnoDB: 3) If the file system or the disk is broken, and you cannot remove
InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
InnoDB: and force InnoDB to continue crash recovery here.
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 386.
[root@ads-dev1 ~]#

I've checked permissions on the files but couldn't find a problem, I also tried running the command using mysql user and got the same error and even ran the next command:

# ulimit -n 4096 ; innobackupex /root/db

But got the exact same error, any ideas?

Itai Ganot
  • 10,644
  • 29
  • 93
  • 146

4 Answers4

3

From doing a little research on error code 24, this seems to be a limit in place on the number files any one user can have open.

You can find these limits by running ulimit -Hn and ulimit -Sn. the -Hn is the unpassable limit, set by root.

Referencing http://www.percona.com/forums/questions-discussions/percona-xtrabackup/8243-innobackupex-1-5-1-error-in-backup

You will have to up the limit of open files until the error goes away. As root user, first: ulimit -a to find the current maximum number of open files, then set a new limit with ulimit -n 2048 / ulimit -n 4094 etc, probably incrementing to the power of 2 until you find the right size for your implementation.

Alex Berry
  • 2,307
  • 13
  • 23
  • I would keep incrementing until you find a limit that works, please take a look at the reference link, the user had to try several different numbers. – Alex Berry Jul 22 '14 at 09:37
  • I kept incrementing even before, but now it works with ulimit -n 90000 – Itai Ganot Jul 22 '14 at 09:47
  • If you take a look here http://www.cyberciti.biz/tips/linux-procfs-file-descriptors.html you might be able to adapt some of it to figure out why you need a 90000 file limit for xtrabackup. – Alex Berry Jul 22 '14 at 10:02
  • Nice, seems like there are 40741 opened files related to mysql's pid. – Itai Ganot Jul 22 '14 at 10:11
0

The solution is very simple, you can pass the argument open-files-limit directly to the xtrabackup command (running as root):

# xtrabackup --backup --parallel=10 --target-dir=... --open-files-limit=100000
the_nuts
  • 430
  • 6
  • 19
0

For this issue, please increase ulimit. First check ulimit -a,ulimit -Sn,ulimit -Hn Then increase limit using by ulimit -n 10240 In my case ,it is working fine Xtrabackup command after changing ulimit

user436681
  • 11
0

Try setting the max open file limit of MySQL too (with ulimit -n 100000):

[mysqld]
open-files-limit = 100000

(don't forget to restart MySQL)

Yvan
  • 412
  • 4
  • 9