o2cb thinks ocfs2 cluster is still online, and refuses to shut down

Question

I have a handful of OpenSuSE 11.2 servers that utilize OCFS2 volumes. I've noticed that o2cb can't figure out when the OCFS2 cluster is actually mounted. For example, when I try to shutdown o2cb, after stopping OCSF2, o2cb refuses to shutdown because it thinks OCFS2 is still up!

After stopping OCFS2 I try to stop o2cb...

hamguy:/dev/disk/by-label # /etc/init.d/o2cb stop
Stopping O2CB cluster ocfs2: Failed
Unable to stop cluster as heartbeat region still active

So I check the status...

hamguy:/dev/disk/by-label # /etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active

And double check OCFS2...

hamguy:/dev/disk/by-label # /etc/init.d/ocfs2 status
Configured OCFS2 mountpoints:  /u/conf /u/logs /u/backup /u/client /u/data /u/mdata

OCFS2 is clearly down, while o2cb clearly thinks otherwise.

The versions of OCFS2 and o2cb are...

kendall@hamguy:~> rpm -qa |grep ocfs2
ocfs2console-1.4.1-25.6.x86_64
ocfs2-tools-o2cb-1.4.1-25.6.x86_64
ocfs2-tools-1.4.1-25.6.x86_64

kendall@hamguy:~> rpm -qa |grep o2cb
ocfs2-tools-o2cb-1.4.1-25.6.x86_64

What causes this, and is there a way around it? If I try to reboot the machine, it will just sit there forever until your physically power cycle it. That obviously is a bit of a problem.

Any insight is appreciated, thank you.

Kendall

Just to verify, does `mount | grep ocfs` and `cat /proc/mounts | grep ocfs` indicate that the OS thinks anything is mounted via ocfs? And what OCFS processes are still showing from `ps -ef | grep ocfs`? — Christopher Karel, Apr 27 '11 at 19:24
I'll try to check that as well, tonight, when I can do it without impacting service. I know I checked 'mount |grep ocfs2' but I didn't copy the output and don't remember what it said. — Kendall, Apr 27 '11 at 19:30

score 2 · Answer 1 · edited Dec 24 '12 at 10:16

umount -l /dev/device 
/etc/init.d/o2cb offline ocfs2

make sure that Checking O2CB heartbeat: is INACTIVE when you do /etc/init.d/o2cb status

if is doesnt change to INACTIVE then remove all its entries from /etc/fstab and reboot.

Check status again after reboot , if it is INACTIVE then unload and re-enable the nodes using:

/etc/init.d/o2cb unload
/etc/init.d/o2cb enable

POINT is the nodes must all be in INACTIVE state before joinging them all to the same heartbeat

IT WORKED FOR ME

o2cb thinks ocfs2 cluster is still online, and refuses to shut down

1 Answers1