2

One of my Mysql replication servers has rebooted during the night and i'm trying to find the reason. Running the command last shows this line: reboot system boot 2.6.32-71.29.1.e Wed Jul 31 04:37 - 11:02 (06:24) I'm trying to read the vmcore file which has been created during the reboot. I've installed kernel-debuginfo package which created the vmlinux file. then I ran:

[root@dbs ~]# crash -s /usr/lib/debug/lib/modules/2.6.32-358.14.1.el6.centos.plus.x86_64/vmlinux /var/crash/127.0.0.1-2013-07-31-01:33:32/vmcore
crash 6.1.0-1.el6
crash: invalid kernel virtual address: 68f8e06b73fe5f1f  type: "possible"
WARNING: cannot read cpu_possible_map
crash: invalid kernel virtual address: fce96c6b73fe5f1f  type: "present"
WARNING: cannot read cpu_present_map
crash: invalid kernel virtual address: e10de07591befc69  type: "online"
WARNING: cannot read cpu_online_map
crash: /usr/lib/debug/lib/modules/2 and /var/crash/127.0.0.1-2013-07-31-01:33:32/vmcore do not match!

Usage:

  crash [OPTION]... NAMELIST MEMORY-IMAGE  (dumpfile form)
  crash [OPTION]... [NAMELIST]             (live system form)

Enter "crash -h" for details.
[root@dbs ~]#

Can anyone please point me to what I'm doing wrong? I wonder how come vmcore and vmlinux do not much as the vmlinux was generated when I installed the kernel-debuginfo package...

[root@dbs 2.6.32-358.14.1.el6.centos.plus.x86_64]# rpm -qa |grep kernel-debug
kernel-debuginfo-common-x86_64-2.6.32-358.14.1.el6.centos.plus.x86_64
kernel-debug-2.6.32-358.14.1.el6.x86_64
kernel-debuginfo-2.6.32-358.14.1.el6.centos.plus.x86_64
[root@dbs 2.6.32-358.14.1.el6.centos.plus.x86_64]# uname -r
2.6.32-71.29.1.el6.x86_64
[root@dbs 2.6.32-358.14.1.el6.centos.plus.x86_64]#

Why did the kernel-debuginfo package generated the wrong vmlinux? uname -r shows kernel version 2.6.32-71.29.1 while the vmlinux which has been generated is of version 2.6.32-358.14.1 ?

[root@dbs ~]# yum list installed |grep kernel
abrt-addon-kerneloops.x86_64
dracut-kernel.noarch    004-33.2.el6_0  @updates                                
kernel.x86_64           2.6.32-71.el6   @anaconda-CentOS-201106060106.x86_64/6.0
kernel.x86_64           2.6.32-71.29.1.el6
kernel-debuginfo.x86_64 2.6.32-358.14.1.el6.centos.plus
kernel-debuginfo-common-x86_64.x86_64
kernel-headers.x86_64   2.6.32-71.29.1.el6
[root@dbs ~]#
Itai Ganot
  • 10,644
  • 29
  • 93
  • 146

1 Answers1

1

I found the solution! The steps I did:

[root@dbs ~]# uname -r
2.6.32-71.29.1.el6.x86_64
[root@dbs ~]#

I ran:

[root@dbs ~]# yum provides kernel-debuginfo-2.6.32

And manually looked for: 

kernel-debuginfo-2.6.32-71.29.1.el6.x86_64

I then ran: rpm -qa |grep ^kernel and yum remove'ed the current installed wrong version of kernel-debuginfo.

Then I installed the correct kernel-debuginfo-2.6.32-71.29.1.el6.x86_64, ran updatedb and then locate'd the new vmlinux with the correct version. Then I used the next command: crash /usr/lib/debug/lib/modules/2.6.32-71.29.1.el6.x86_64/vmlinux /var/crash/127.0.0.1-2013-07-31-01\:33\:32/vmcore and the crash report ran successfully. Thanks for your help @ALex_hha .

Itai Ganot
  • 10,644
  • 29
  • 93
  • 146
  • I am having same issue. I checked the kernel version and the debug info kernel package version are same, however, I am still getting the error crash: page excluded: kernel virtual address: ffffffff80488620 type: "present" WARNING: cannot read cpu_present_map crash: page excluded: kernel virtual address: ffffffff804832e0 type: "online" WARNING: cannot read cpu_online_map crash: page excluded: kernel virtual address: ffffffff8048a240 type: "xtime" – chandank Aug 20 '13 at 20:42
  • Does the running kernel need to be the debug kernel? If I run the debug kernel and then panic it, I am able to get it work with the dump, however, if I run the normal kernel and then panic it to get the dump it does not work – chandank Aug 20 '13 at 21:27
  • I used it on a regular kernel, not a debug one – Itai Ganot Aug 21 '13 at 06:55