Linux CentOS: A server throws weird error messages, can you speculate why?

Question

There's a server in my company which is part of the production environment. On the server there's an ActiveMQ server running. I logged into the Active-MQ UI and tried to create a new queue. When I did, I got this message:

HTTP ERROR: 500

/workspace/development/org/apache/activemq/5.1.0/data/kr-store/data/data-container-roots-2 (Read-only file system)
RequestURI=/admin/createDestination.action

Caused by:

java.io.FileNotFoundException: /workspace/development/org/apache/activemq/5.1.0/data/kr-store/data/data-container-roots-2 (Read-only file system)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.(RandomAccessFile.java:212)
    at org.apache.activemq.kaha.impl.data.DataFile.getRandomAccessFile(DataFile.java:51)
    at org.apache.activemq.kaha.impl.data.SyncDataFileWriter.storeItem(SyncDataFileWriter.java:71)

I'm aware to the "File not found" message, but it seems like it's not related directly to the issue.

In order to troubleshoot the issue, I logged into the server and ran some tests, during these tests I found that some basic commands which i'm trying to run are failing with the same error:

[root@ctrl3 kr-store]# touch 1
touch: cannot touch `1': Read-only file system
[root@ctrl3 /]# chgrp users /workspace
chgrp: changing group of `/workspace': Read-only file system
[root@ctrl3 kr-store]# chown peeradmin.users /workspace
chown: changing ownership of `/workspace': Read-only file system
[root@ctrl3 kr-store]# ls -ld data
drwxrwxr-x 2 peeradmin users 4096 AUG 12 12:27 data
[root@ctrl3 kr-store]# chmod o+w data/
chmod: changing permissions of `data/': Read-only file system

If I remember correctly, the last time I encountered such an error, we found later that there was a I/O issue with the disk, but if that's not the case, what else could it be?

Edit #1:

[root@ctrl3 kr-store]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 ro,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
/sys /sys sysfs rw 0 0
/proc/bus/usb /proc/bus/usb usbfs rw 0 0
devpts /dev/pts devpts rw 0 0
/dev/sda7 /tmp ext3 rw,data=ordered 0 0
/dev/VolGroup00/LogVol00 /workspace ext3 ro,data=ordered 0 0
/dev/sda5 /usr ext3 rw,data=ordered 0 0
/dev/sda3 /var ext3 rw,data=ordered 0 0
/dev/sda1 /boot ext3 rw,data=ordered 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
/etc/auto.misc /misc autofs rw,fd=7,pgrp=3795,timeout=300,minproto=5,maxproto=5,indirect 0 0
-hosts /net autofs rw,fd=13,pgrp=3795,timeout=300,minproto=5,maxproto=5,indirect 0 0
atlas.sj.company.com:/volumes/atlas_vol/NFS1 /nfs1 nfs rw,noatime,vers=3,rsize=32768,wsize=32768,soft,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=atlas.sj.company.com 0 0
atlas.sj.company.com:/volumes/atlas_vol/NFS1/NIS/home /home nfs rw,noatime,vers=3,rsize=32768,wsize=32768,soft,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=atlas.sj.company.com 0 0
atlas.sj.company.com:/volumes/atlas_vol/NFS1 /nfs1 nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=atlas.sj.company.com 0 0
atlas.sj.company.com:/volumes/atlas_vol/NFS1/NIS/home /home nfs rw,noatime,vers=3,rsize=1048576,wsize=1048576,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=atlas.sj.company.com 0 0

Sven: The log says nothing:

[root@ctrl3 kr-store]# cat /var/log/messages |grep -v [xinetd\|snmpd]
[root@ctrl3 kr-store]#

Also, if nothing can be written to the disk then I guess that the logs cannot be updated as well.

Edit #2: So it seems like the file system have gone corrupted somehow... am I right?

SCSI device sdb: 1953525168 512-byte hdwr sectors (1000205 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
ext3_abort called.
EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
sd 1:0:0:0: SCSI error: return code = 0x06000000
end_request: I/O error, dev sdb, sector 745962211
printk: 215 messages suppressed.
Buffer I/O error on device dm-0, logical block 51773423
lost page write due to I/O error on dm-0
Buffer I/O error on device dm-0, logical block 51773424
lost page write due to I/O error on dm-0
Buffer I/O error on device dm-0, logical block 51773425
lost page write due to I/O error on dm-0

Thanks in advance,

For some reason they file system hosting `/workspace` is mounted read-only. What does the log files say? — Sven, Jan 20 '15 at 14:14
@etagenklo, please check Edit #2, it seems like one of the file systems have gone corrupted... I wonder why and how. — Itai Ganot, Jan 20 '15 at 14:27

etagenklo · Accepted Answer · 2015-01-20T14:28:12.457

Your filesystem seems to be mounted read-only. You can check this via cat /proc/mounts. A filesystem getting remounted read-only is usually caused by filesystem errors. The reason for this can be harddisk problems, so you should check your disks (SMART values, controller status in case of a HW RAID etc.)

Edit #1: Your mounts show that it's indeed mounted read-only:

/dev/VolGroup00/LogVol00 /workspace ext3 ro,data=ordered 0 0

What you can try is to remount the Volume as writeable again, but I wouldn't recommend this before you found out why it was remounted read-only before, otherwise you'll risk losing data:

mount -o remount,rw /workspace

You should in any case first check the output of dmesg and check your hdds smart status via smartctl.

Edit #2:

Seems like sdb is the physical problem here:

end_request: I/O error, dev sdb, sector 745962211

Check the output of

smartctl -a /dev/sdb

Linux CentOS: A server throws weird error messages, can you speculate why?

1 Answers1