MySql Server does not start with innodb_force_recovery set to value 1-6

Question

I have an ubuntu 14.04 KVM that was running as server LAMP till today when I have installed a Wordpress Plugin and the MySql Server 5.5 stopped working.

I have tried to start the server multiple times with innodb_force_recovery set from 1 to 6 so I can recover the data and restore the informations. I do not care to loose some data but I can't afford to loose the whole lot. (I should have done more frequent Backup, my bad).

The configuration of my mysqld is:

[MYSQLD]
innodb_force_recovery=6
innodb_fast_shutdown=0
innodb_purge_threads=0
#table_open_cache=4450
#table_definition_cache=1450
#innodb_buffer_pool_size=4G

The error logs gives me this:

000000000000000000000000000000000000000000000000000000000000000000(the 0 are many more but I did not paste it); asc                                                                                                                                                                                                                                   ## here I have many many empty lines                                                                                                                                                       ;
InnoDB: End of page dump
160428 18:34:39  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
160428 18:34:39  InnoDB: Assertion failure in thread 140591351621376 in file rem0rec.c line 569
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
17:34:39 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346701 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x20)[0x5613594c3390]
/usr/sbin/mysqld(handle_fatal_signal+0x3d5)[0x5613593ac785]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fde0f230340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7fde0e887cc9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fde0e88b0d8]
/usr/sbin/mysqld(+0x6534f0)[0x5613596234f0]
/usr/sbin/mysqld(+0x640f5d)[0x561359610f5d]
/usr/sbin/mysqld(+0x5bcd8e)[0x56135958cd8e]
/usr/sbin/mysqld(+0x57f25d)[0x56135954f25d]
/usr/sbin/mysqld(+0x65bb03)[0x56135962bb03]
/usr/sbin/mysqld(+0x660361)[0x561359630361]
/usr/sbin/mysqld(+0x65c128)[0x56135962c128]
/usr/sbin/mysqld(+0x651ac4)[0x561359621ac4]
/usr/sbin/mysqld(+0x5a1b93)[0x561359571b93]
/usr/sbin/mysqld(+0x5a210e)[0x56135957210e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fde0f228182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fde0e94b47d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
InnoDB: End of page dump
160428 18:34:39  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
160428 18:34:39  InnoDB: Assertion failure in thread 140591351621376 in file rem0rec.c line 569
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
17:34:39 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346701 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x20)[0x5613594c3390]
/usr/sbin/mysqld(handle_fatal_signal+0x3d5)[0x5613593ac785]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fde0f230340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7fde0e887cc9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fde0e88b0d8]
/usr/sbin/mysqld(+0x6534f0)[0x5613596234f0]
/usr/sbin/mysqld(+0x640f5d)[0x561359610f5d]
/usr/sbin/mysqld(+0x5bcd8e)[0x56135958cd8e]
/usr/sbin/mysqld(+0x57f25d)[0x56135954f25d]
/usr/sbin/mysqld(+0x65bb03)[0x56135962bb03]
/usr/sbin/mysqld(+0x660361)[0x561359630361]
/usr/sbin/mysqld(+0x65c128)[0x56135962c128]
/usr/sbin/mysqld(+0x651ac4)[0x561359621ac4]
/usr/sbin/mysqld(+0x5a1b93)[0x561359571b93]
/usr/sbin/mysqld(+0x5a210e)[0x56135957210e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fde0f228182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fde0e94b47d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

My main aim is to recover the DBs but I can not start the server. Somebody can help me to find a solution? I've been trying for hours and I am short of options.

Thank you!

Answer 1

I manage to start the server using #mysqld_safe --innodb_force_recovery 4 . I have tried hundreds of time to restart the mysql server but like this started.

After I restored the backups that I was able to make and everything seems to work properly.

Answer 2

For me, recovery wasn't an option, no matter what level innodb_force_recovery was set to.

Based on this answer, and because InnoDB being part of the trouble, I disabled InnoDB altogether.

You can edit /etc/my.cnf to add what I added, or the innodb_force_recovery=1 statement, but it must be in the section [mysqld], which a grep search found was already in /etc/my.cnf.d/server.cnf. So, there I added to [mysqld]:

[mysqld]
innodb=OFF
default_storage_engine=MyISAM

Then, it started fine and I could get in and start working with MySQL.