0

I'm using Percona 5.7.18-16 for my MySQL (Percona) DB server.

I'm using Percona xtrabackup 2.4.8 to make live backups of the running server.

I've set up a testing server to verify backups made from the above primary.

The testing server is also running Percona 5.7.18-16.

I've found that Percona xtrabackup 2.4.8 almost -never- makes a viable backup... even if it reports "Completed OK!" in the CLI when completing a backup phase. On the testing server, there are about 80% of the time corrupt indices when the backup is corrupt, and sometimes the tables themselves are corrupt when the backup is restored.

Running a "check table tablename" on the verification server crashes the verification server in innodb with various table and index corruptions for each backup.

The DB is about 400GB in size when live. I've ensured via a sha256sum that the .tar I restore on the testing server is binary-exact the same as on the source / main server. I use

rsync -avh --progress --no-whole-file --partial user@source_server/path_to_tar.tar .

to transfer the .tar of the xtrabackup-produced directory to the verification/testing server.

I'm calling it via the innobackupex symlink, e. g.

innobackupex --user=myuser --password=mypass --parallel=4 --rsync ...

and then

innobackupex --apply-log --user=myuser --password=mypass --use-memory=2GB ...

About 90% of the backups produced this way are corrupt, with no errors or failures reported by xtrabackup after the backup is produced. This is an alarming rate of failure for an enterprise application.

Any ideas on what I can try to address this? Xtrabackup appears a common and popular tool.

I have run a "check table ..." on the live server's tables, and it is clean. The 5.7 Percona instance on it is running for about 370 days at this point. It handles about 25 million inserts and about 30 million updates every 24 hours for the past year.

But it is proving imppossible to back up consistently with xtrabackup.

Any pointers? Anybody encounter this before? How can one get xtrabackup to make consistent backups of a Percona 5.7 instance?

Thanks!

Stefan

Stefan
  • 316
  • 2
  • 16
  • How are you determining it to be corrupt? – ceejayoz Sep 23 '19 at 15:22
  • Percona.com should be able to HELP you. Enlist their resources rather than any 3rd party. – Wilson Hauck Sep 23 '19 at 21:54
  • Ran a check table on three critical tables. Each time at least one entire table is reported corrupt, or an index of it. The testing server instance then crashes and logs a crash dump in /var/log/mysqld.log which identifies the corrupt table and which page is corrupt, or lists the index by name that is corrupt. This happens for almost every backup (about 90% are produced corrupt, over and over, by xtrabackup 2.4.8 from official Percona repo.) – Stefan Sep 25 '19 at 06:06

0 Answers0